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EXECUTIVE  SUMMARY 

Purpose — This  report  provides  an  in-depth  analysis  to  assist  future  Command  and  Control  (C2)  decision 
makers  in  determining  the  conditions  necessary  for  effective  distributed  learning  for  future  C2  systems.  A 
synopsis  of  findings  are  presented  in  terms  of  (a)  a  summative  evaluation  that  identifies  strengths,  weak¬ 
nesses,  lessons  learned;  (b)  best  practices  of  the  Theater  Battle  Management  Core  Systems  (TBMCS) 
training  program;  and  (c)  a  holistic  view  (context,  input,  process,  and  product  evaluation)  of  the  TBMCS 
distributed  training  program  that  shows  the  impact  of  training,  not  only  on  the  individual  but  on  the 
United  States  Air  Force  (USAF)  as  well. 

Background — To  met  the  learning  requirements  of  the  future  force,  the  Secretary  of  Defense  stated: 

“DOD  personnel  will  have  access  to  the  highest  quality  training  that  can  be  tailored 
to  their  needs  and  delivered  cost  effectively,  anytime  and  anywhere.  Furthermore  to 
achieve  this  vision  anytime,  anywhere  learning  must  be  distributed,  just-in-time  and 
on-demand  and  enabled  with  resources,  development  and  exploitation  of  learning 
technologies  ”. 

The  Department  of  Defense  (DOD)  Strategic  Plan  for  Advanced  Distributed  Learning  (ADL)  dated  April 
30,1999  identifies  an  ADL  initiative  intended  to  implement  the  Secretary  of  Defense’s  training  vision. 
Electronic  Systems  Center  (ESC)  Combat  Air  Force  Command  and  Control  (CAFC2)  Distributed 
Learning  Initiative — ESC  was  proactive  in  meeting  the  learning  and  technology  needs  identified  in  the 
ADL  initiatives  and  DOD  Strategic  Plan  when  developing  TBMCS  training  material.  A  great  deal  of  pro¬ 
gress  was  made  in  shifting  from  a  paper-based,  instmctor-led  training  program  established  in  1995,  to  a 
distributed  web  based  training  program  led  by  facilitation  upon  fielding  in  2001.  Meeting  the  require¬ 
ments  of  anywhere,  anytime,  and  anyplace  learning  requires  solutions  to  many  technical,  security,  and 
financial  barriers.  As  users  from  locations  worldwide  attempted  to  access  TBMCS  materials  located  on 
distributed  servers,  three  problems  emerged.  The  first  problem  was  accessing  materials  from  remote  loca¬ 
tions;  the  second  was  NIPRNET  bandwidth;  and  the  third  was  local  computer  security  initiatives  hinder¬ 
ing  the  use  of  web  servers  at  user  locations. 

Data  Collection  Techniques — Due  to  the  numerous  training  baseline  changes  conducted  during  the  over¬ 
all  software  development  evolution,  inconsistencies  of  data  collection,  and  lack  of  raw  data,  this  study  did 
not  lend  itself  to  a  hypothesis  testing  approach.  Instead,  an  exploratory  research  methodology  was  chosen 
to  support  Systems  Program  Director  (SPD)  concerns.  Seven  research  questions  presented  in  this  report 
were  identified  by  the  SPD  as  the  basis  for  determining  the  effectiveness  of  the  TBMCS  distributed 
training  program.  Kirkpatrick’s  theory  of  evaluation  [satisfaction,  learning,  transfer,  and  Return  on  In- 
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vestment  (ROI)]  was  used  to  categorize  the  data  collected.  Overall  data  was  gathered  using  four  collection 
methods:  surveys,  focus  groups,  pre -/post-tests,  and  a  student  self-assessment. 

Findings — A  major  goal  of  this  study  is  to  determine  the  impact  of  training.  For  purposes  of  this  study 
impact  was  viewed  as  “measurable  learning”  and  “student  perception”  of  learning.  Spiral  development 
encourages  user  participation  and  involvement  and  assessment  of  software  and  training  development.  A 
major  emphasis  placed  on  military  and  industry  training  evaluations  are  student  reactions  known  as  “hap¬ 
piness  indicators”.  These  are  categorized  as  user  perception.  Perception  drives  motivation  and  emotion. 
Emotion  is  often  a  more  powerful  influence  on  behavior  than  logic  or  empirical  data.  Thus,  it  is  an  m- 
portant  indicator  of  course  satisfaction.  End  of  course  critiques  suggested  that  students  were  not  “satis¬ 
fied”  with  the  concept  of  self-paced  distributed  learning  with  little  human  interaction;  however, 
measurable  pre-/post-test  scores  revealed  that  students  understood  facts  and  concepts,  suggesting  that 
knowledge  “achievement”  resulted  from  the  training.  Eocus  groups  revealed  that  students  perceived 
training  would  be  greatly  enhanced  if  the  implementation  approach  reverted  back  to  the  traditional  use  of 
mobile  training  teams.  This  requirement  was  identified  to  the  Training  Planning  Team  (TPT)  for  valida¬ 
tion,  and  ESC  was  requested  to  shift  from  a  distributed  learning  environment  back  to  a  traditional,  h- 
stmctor-  led,  “hands-on”  approach  to  training.  The  change  in  scope  was  a  costly  decision  to  the  SPD. 
Recommendations  for  Future  ADL  Training — Seven  barriers  are  identified  as  impediments  to  a  success¬ 
ful  implementation  of  the  TBMCS  distributed  learning  program.  They  are:  inconsistent  funding,  change 
of  training  requirements,  lack  of  established  evaluation  criteria,  inconsistent  On-the-Job-Training  (OJT) 
programs  after  fielding,  lack  of  technology  planning,  unknown  factors  for  determining  ROI,  lack  of  local 
distance  learning  policy  and  management  enforcement,  and  lack  of  awareness  and  understanding  of 
changing  roles  and  responsibilities  for  students  and  instmctors  in  distance  learning  environments.  These 
can  be  overcome  if  known  in  advance  by  the  SPD  and  emphasis  is  placed  on  establishing  processes  to 
overcome  these  barriers. 
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1.0— INTRODUCTION 


1.1  PURPOSE 

DODI  5000.2,  The  Defense  Acquisition  System  states  that  the  SPD  shall  ensure  that  the  design  and 
acquisition  of  systems  will  be  cost  effectively  supported  and  shall  ensure  that  these  systems  are  provided 
to  the  user  with  the  necessary  support  infrastmcture  for  achieving  the  user’s  peacetime  and  wartime 
readiness  requirements.  Support  resources  include  operator  and  maintenance  manuals,  tools,  equipment, 
and  training.  Furthermore,  the  SPD  shall  consider  the  use  of  embedded  training  and  maintenance  tech¬ 
niques  to  enhance  user  capability  and  reduce  life  cycle  costs.  Air  Force  Instruction  (AFI)  63-123,  Evob- 
tionary  Acquisition  for  Combat  and  Control  Systems,  states  that  prior  to  system  fielding,  the  SPD  shall 
ensure  sufficient  training  is  complete  to  fulfill  approved  operational  concepts  of  employment  and  suffi¬ 
cient  support  in  place  to  fix  failures  and  sustain  the  system. 

In  accordance  with  (lAW)  the  policies  stated  above,  the  SPD  for  TBMCS  ESC,  CAEC2  is  responsi¬ 
ble  for  managing,  preparing,  and  conducting  training  for  the  TBMCS  system  as  it  is  fielded.  How  does 
the  SPD  ensure  the  adequacy  of  TBMCS  training?  What  criteria  is  TBMCS  training  evaluated  against? 
What  data  should  be  collected?  AFI  36-2211  identifies  the  Instmctional  Systems  Development  (ISD)  pro¬ 
cess  as  a  systematic  approach  to  developing  and  conducting  training.  The  ISD  process  includes  five 
phases — analysis,  design,  development,  implementation,  and  evaluation.  ESC  utilized  this  process  in  the 
development  of  TBMCS  training  materials.  Worthen,  Sanders,  and  Eitzpatrick  (1997)  stated  that  SPDs 
face  the  following  kinds  of  education  and  training  evaluation  considerations: 

•  Context  evaluations  that  serve  as  planning  decisions  to  determine  what  needs  are  to  be  ad¬ 
dressed 

•  Input  evaluations  to  serve  stmcturing  decisions  in  determining  what  resources  are  available  and 
what  training  strategies  should  be  considered 

•  Process  evaluations  to  serve  as  implementing  decisions  such  as  how  well  the  plan  is  being  im¬ 
plemented  and  what  barriers  threaten  its  success 

•  Product  evaluations  to  serve  future  product  decisions 
The  goals  of  this  paper  are  to  provide: 

•  A  summative  evaluation  that  identifies  strengths,  weaknesses,  lessons  learned,  and  best  prac¬ 
tices  of  the  TBMCS  training  program 

•  An  in-depth  analysis  in  assisting  future  Command  and  Control  (C2)  decision  makers  in  deter¬ 
mining  under  what  conditions  distributed  learning  is  likely  to  be  effective  for  future  C2  systems 

•  A  holistic  view  (context,  input,  process,  product  evaluation)  of  TBMCS  training  that  shows  the 
impact  of  training,  not  only  on  the  individual  but  on  the  USAE  as  well 
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2.0— DEFINITIONS 

Adult  Learner.  One  who  (1)  performs  social  roles  that  our  culture  assigns  to  adults  (e.g.,  worker,  spouse, 
soldier,  responsible  citizen),  and  (2)  perceives  himself  to  be  responsible  for  his  life. 

Advanced  Distributed  Learning  Initiative.  The  DOD  strategy  for  using  learning  and  information  tech¬ 
nologies  to  modernize  education  and  training.  A  key  requirement  is  the  ability  to  reuse  instructional  com¬ 
ponents  in  multiple  applications  and  environments  regardless  of  the  tools  used  to  create  them. 

Distanced  learning .  A  general  term  used  to  cover  the  broad  range  of  teaching  and  learning  events  in 
which  the  student  is  separated  from  the  instmctor,  or  other  students,  by  distance  and/or  time. 

Learning .  A  change,  or  the  capacity  to  change,  one’s  level  of  ability  or  knowledge. 

Spiral  Development.  Software  development  spirals  facilitate  more  precise  and  rapid  maturation  of  new 
technologies  and  refinement  of  user  requirements  with  high  operational  utility  into  a  complete  capability 
for  one  increment.  The  key  intent  is  for  the  system  and  the  fidelity  of  its  requirements  to  evolve  together 
with  iterative  feedback. 

Type-1  Training.  Contract  training  or  factory  training  that  Air  Education  Training  Command  (AETC) 
arranges  for  Air  Eorce  and  other  DOD  personnel  and  contractors  to  conduct  at  either  the  contractor’s  b- 
cation  or  a  DOD  facility. 

Legacy  Systems.  This  term  refers  to  the  older  Air  Eorce  software  applications  that  TBMCS  replaced. 
Major  systems  consisted  of  Wing  Command  and  Control  System  (WCCS),  Combat  Intelligence  System 
(CIS),  and  Contingency  Theater  Air  Planning  System  (CTAPS). 

Mobile  Training  Team.  A  cadre  of  instructors  who  travel  to  military  locations  to  provide  TBMCS  train¬ 
ing  to  users  of  the  system. 

Multi-service .  For  purposes  of  this  study,  multi-service  refers  to  the  US  Navy,  Air  Force,  IVh- 
rines,  and  North  American  Aerospace  Defense  (NORAD)  personnel  who  utilize  TBMCS. 

TBMCS.  Provides  automated  C2  and  decision  support  tools  to  improve  the  planning,  preparation,  and 
execution  of  joint  air  combat  capabilities. 
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3.0— ORGANIZATIONAL  ACTIVITIES 

The  creation  of  a  distributed  learning  environment,  which  can  support  multi-service  requirements  of 
anywhere,  anytime,  and  anyplace  learning,  requires  solutions  to  many  technical,  legal,  security,  and  fi¬ 
nancial  barriers.  To  overcome  these  barriers  and  to  develop  and  implement  standards,  numerous  DOD 
organizations  have  instigated  distance  learning  efforts.  Some  of  the  DOD  organizations  and  their  respec¬ 
tive  projects  are  described  below. 

3.1  SECRETARY  OF  DEFENSE 

Executive  Order  13111  was  released  January  1999,  which  required  DOD  to  provide  flexible  learning 
opportunities,  use  technology  to  improve  training  opportunities,  and  place  emphasis  on  lifelong  learning. 
The  creation  of  an  advanced  information  infrastructure,  which  can  support  an  educational  environment  on 
a  national  basis,  requires  solutions  to  many  technical,  legal,  security  financial,  and  regulatory  barriers. 

To  met  the  learning  requirements  of  the  future  force,  the  Secretary  of  Defense  Isted  the  following 
training  technology  vision  in  The  Report  to  the  106“’  Congress; 

“To  ensure  that  DOD  personnel  have  access  to  the  highest  quality  education  and 
training  that  can  be  tailored  to  their  needs  and  delivered  cost  effectively,  anytime 
and  anywhere  ”.  To  achieve  this  vision  anytime,  anywhere  learning  must  be  dis¬ 
tributed,  just-in-time  and  on-demand  and  enabled  with  resources,  development 
and  exploitation  of  learning  technologies'". 

3.2  JOINT  VISION  (JV)  2010  AND  2020 

JV  2010  provides  the  conceptual  template  for  how  we  will  channel  the  vitality  of  our  people  and  lev¬ 
erage  technological  opportunities  to  achieve  new  levels  of  effectiveness  in  joint  warfighting.  Specifically, 
the  armed  forces  of  the  future  must  be  able  to  fight  in  joint,  combined,  and  interagency  environments  en¬ 
abled  by  information  superiority  -  the  gathering,  processing,  fusion,  and  dissemination  of  more  accurate 
and  timely  information  and  knowledge,  anywhere,  anytime  and  every  time.  Our  future  forces  must  be 
highly  adaptive,  learning  forces  that  organize  to  meet  threats  effectively  and  rapidly.  They  must  continu¬ 
ously  learn,  simulate,  and  rehearse,  whether  they  are  in  school,  at  home  station,  at  home,  en  route  to  or  in 
the  theater  of  operations.  Providing  anytime  anywhere  training  is  a  key  to  maintaining  military  readiness 
in  the  information  age  and  is  one  of  our  foremost  priorities  (Shalikashvili,  1996).  JV2020  requires  what 
the  military  must  apply  to  achieve  full  spectrum  dominance  as  introduced  in  JV2010  and  focuses  on  three 
factors  (interoperability,  innovation,  and  decision  superiority)  as  central  to  success  in  the  four  original 
cornerstone  JV2010  operational  concepts  of  dominant  maneuver,  precision  engagement,  focused  logistics, 
and  full  dimensional  protection.  Training  anytime,  anywhere,  and  anyplace  will  continue  to  be  the  key  to 
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enable  commanders  to  make  better  and  faster  decisions  than  their  opponents. 
(http://www.defenselink.mil/news) 

3.3  DOD  ADVANCE  DISTRffiUTED  LEARNING  INITIATIVE 

The  DOD  Strategic  Plan  for  ADL  dated  April  30,1999  identifies  the  DOD  ADL  initiative.  This  ini¬ 
tiative  sets  forth  a  new  paradigm  intended  to  implement  the  Secretary  of  Defense’s  training  vision.  In 
short,  the  strategy  is  to:  pursue  emerging  network-based  technologies;  create  common  standards  that  will 
enable  reuse  and  interoperability  of  learning  content;  lower  development  costs;  promote  widespread  col¬ 
laboration  that  can  satisfy  common  needs;  enhance  performance  with  next-generation  learning  technob- 
gies;  work  closely  with  industry  to  influence  Commercial-Off-the-Shelf  (COTS)  product  development 
cycle;  and  establish  a  coordinated  implementation  process.  The  following  five  elements  are  needed  to  de¬ 
velop  and  successfully  implement  ADL: 

•  common  industry  standards 

•  interoperable  tools  and  content 

•  a  robust  and  dynamic  network  infrastructure  for  distribution 

•  supporting  resources 

•  culmral  change  at  all  levels  of  command,  recognizing  that  learning  is  an  official  requirement  of 
the  duty  day 

3.4  DEFENSE  ACQUISITION  UNIVERSITY  (DAU) 

The  DAU  is  committed  to  providing  high-quality  education  and  training  to  the  members  of  the  DOD 
acquisition  community.  Emerging  technologies  provide  DAU  with  the  ability  to  increase  access  to  its 
courses  while  promoting  effective  learning  experiences  for  individuals.  DAU’s  overall  curriculum  transi¬ 
tion  strategy  is  based  on  the  ADL  mandate  to  use  technology  to  deliver  quality  training  to  learners  in  a 
cost-effective  way.  To  meet  the  mandate,  DAU  identified  10%  of  DAU  courses  to  be  converted  to  the  use 
of  information  age  technologies  before  the  end  of  FY97  and  an  additional  15%  by  the  end  of  FY98  (John¬ 
son  1997).  To  assist  the  Acquisition  Workforce  in  meeting  the  requirements  of  this  continuous  learning 
policy,  in  1999  DAU  created  an  online  Continuous  Learning  Center  (CLC).  The  CLC,  through  a  single 
point  of  access,  offers  continuous  learning  modules;  regulation,  policy,  and  guidance  resources;  and  col¬ 
laboration  tools  that  provide  expertise  from  peers  and  other  professionals.  It  is  designed  around  the  meta¬ 
phor  of  a  campus  map,  which  links  to  buildings  that  house  functions  one  would  typically  find  on  a 
campus  (http://clc.dau.mil). 
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3.5  US  ARMY 

In  the  past  decade,  the  Armed  forces  experienced  significant  force  adjustments  and  resource  reduc¬ 
tions.  Army  forces  were  tasked  to  deploy  for  operational  missions  that  varied  from  providing  humanitar¬ 
ian  relief  to  engaging  in  major  conflicts  such  as  Operation  Desert  Storm.  More  reliance  was  placed  on  the 
reserve  components  to  support  the  active  component.  Additionally,  state-of-the-art  digital  equipment  was 
fielded.  This  combination  of  a  smaller  army  operating  with  new  technology  to  accomplish  expanded  mis¬ 
sions,  dictated  the  upgrading  of  current  training  methods  and  procedures  to  ensure  a  combat-ready  force. 
To  meet  these  needs  the  Total  Army  Distance  Learning  Program  (TADLP) 
(http:/www.tadlp.monroe.army.mil/)  was  created  The  TADLP  system  consists  of  three  different  distance 
learning  facilities  connected  by  a  worldwide  communications  network.  (US  Army  Training  and  Doctrine 
Command,  2001). 

3.6  US  MARINE  CORPS 

The  transition  to  Interactive  Multimedia  Instmction  (IMI)  has  wide-reaching  implications  across  the 
Marine  Corps.  The  increasing  reliance  on  distance  learning  to  meet  broad  training  requirements  addresses 
critical  problem  areas  including  the  limited  capacity  of  schools,  outdated  paper-based  training  materials, 
and  the  expense  of  sending  active-duty  Marines  to  formal  schools  for  specialized  and  professional  mili¬ 
tary  education  training.  As  a  key  component  of  the  Marine  Corps  Training  Modernization  Initiative, 
MarineNet  helps  transform  the  way  training  and  education  are  delivered  to  active  and  reserve  Marines 
around  the  world.  The  MarineNet  program  establishes  both  a  distributed  learning  network  infrastmcture 
and  a  program  for  development  of  IMI.  To  build  this  distributed  learning  infrastmcture,  the  Marine  Corps 
is  rapidly  investing  significant  resources  towards  upgrading  telecommunications  and  instmctional  facili¬ 
ties.  (Tyler  and  Harper,  1999). 

3.7  US  NAVY 

The  Navy  distance  learning  vision  is  to  deliver  training  on  demand,  to  the  right  people  to  meet  rap¬ 
idly  changing  mission  tasks  in  the  increasingly  complex,  network-centric  warfare  environment.  Navy 
training  will  be  viewed  as  a  career  long  learning  continuation  that  involves  technical  skills  and  warrior 
training,  and  professional  and  leadership  development  and  education.  Availability  and  diversity  of  profes¬ 
sional  and  continuous  education  will  increase  significantly.  The  training  and  education  community  will 
leverage  off  the  Navy’s  enterprise/corporate  information  network,  to  deliver  training  and  education  to  the 
entire  force,  active  duty  and  reserve,  anywhere  and  anytime.  Education  for  the  officer,  enlisted,  and  civil¬ 
ian  workforce  will  be  provided  through  a  “Virtual  Navy  University”  providing  an  array  of  Navy-specific 
and  professional  development  curricula.  The  results  produced  from  the  Navy’s  restmctured  training  and 
education  system  will  be  a  technically  skilled  and  agile  workforce  and  network-centric  warriors  who  are 
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prudent  risk  takers  and  innovators;  adaptive  and  continuous  learners;  and  effective  leaders,  mentors,  and 
teachers  (http;//www.ott.navv.mil/). 

3.8  US  AIR  FORCE 

The  Air  Force  Distance  Learning  Office  (AFDLO),  located  at  Maxwell  AFB,  was  established  as  the 
focal  point  within  the  Air  Force  for  implementation  of  Distance  Learning  (DL)  policy  and  emerging  tech¬ 
nology  (http;//www.au.af.mil/au/schools/afiadl.html).  As  the  hub  for  Air  Force  DL,  the  office  pro¬ 
vides  policy  and  guidance,  as  well  as  consultation  and  support  for  the  planning  and  development  of 
distance  learning  programs.  Air  Force  distance  learning  goals  are  to: 

•  create  an  environment  that  recognizes  the  value  of  distance  learning 

•  ensure  availability  of  resources  to  meet  education  and  training  requirements 

•  ensure  Total  Force  interoperability  for  all  distance  learning  instmctional  technology 

•  capitalize  on  appropriate  leading  edge  technology 

•  improve  educational  and  training  efficiencies  where  practical  and  cost  effective 

3.9  ESC  CAFC2  INITIATIVES  TO  SUPPORT  THE  ADL  INITIATIVE 

ESC  CAFC2  was  proactive  in  meeting  the  learning  and  technology  needs  identified  in  JV2010  as 
well  as  the  ADL  initiatives  in  the  DOD  Strategic  Plan  when  developing  TBMCS  training  material.  Much 
progress  was  made  in  shifting  from  a  paper-based  instmctor  led  training  program  established  in  1995,  to  a 
distributed  web-based  training  program  led  by  facilitation  upon  fielding  in  2001.  TBMCS  training  materi¬ 
als  were  developed  using  COTS  technology  and  are  interoperable/platform  independent  in  a  Unix  and  PC 
environments.  The  ESC  CAEC2  training  development  contractor  maintained  unclassified  Hypertext 
Markup  Language  (HTML)  training  material  covering  TBMCS  operations  on  a  stand-alone  web  server. 
Eight  additional  servers  were  built  as  “clones”  of  the  master  server.  The  cloned  servers  were  dispersed  to 
military  installations  worldwide  and  act  as  stand-alone  regional  web  servers.  Each  regional  web  server  is 
connected  to  the  web  through  the  local  host  installations  network  configuration  architecture. 

In  April  2000,  an  electronic  Computer  Managed  Instmction  (CMI)  software  application  was  pur¬ 
chased  and  distributed  to  the  cloned  servers  to  track  student  progress/completion  of  the  distributed  train¬ 
ing.  The  CMI  program  organizes  the  online  training  lessons  through  a  learner-profile  customized  for  each 
student  based  on  their  course  registration.  Completion  statuses  of  individual  training  lessons  are  main¬ 
tained  in  the  database  and  accessible  to  the  individual  students,  supervisors,  or  training  administrators. 
Student  records  are  replicated  to  the  cloned  servers  through  database  replication  protocols. 
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3.10  CORRELATION  OF  DOD  AND  ESC  CAFC2  INITIATIVES  RELATING  TO  THE  TBMCS 
TRAINING  REPORT 

The  ESC  CAFC2  TBMCS  training  division  is  on  the  leading  edge  of  utilizing  the  most  current  tech¬ 
nology  available  for  designing  and  developing  training  for  TBMCS.  During  this  period  of  technology  in- 
plementation  ESC  CAEC2  needs  to  ensure  compliance  with  current  policy,  newly  developed  standards, 
and  established  COTS  products.  ESC  CAEC2  has  the  responsibility  to  ensure  the  distributed  training  is 
effective.  Zane  and  Mrzowski  (2001)  stated  the  research  in  distance  education  conducted  to  date  has  re¬ 
ceived  harsh  and  consistent  criticism.  Criticism  often  focuses  on:  (1)  lack  of  control  for  extraneous  vari¬ 
ables,  (2)  lack  of  use  of  randomly  selected  subjects,  (3)  lack  of  validity  and  reliability  of  the  instmments 
used  to  measure  student  outcomes,  and  (4)  inadequate  control  or  the  feelings  and  attitudes  of  the  students 
and  faculty.  Champagne,  Pawluk,  Wisher,  and  Cumow  (1999)  stated  that  published  literature  on  the  ef¬ 
fectiveness  of  distance  learning  is  overwhelmingly  anecdotal.  A  review  of  the  approximately  200  papers 
published  in  the  1996-1998  proceedings  of  the  annual  DL  conference  found  only  8%  concerned  with  em¬ 
pirical  studies  of  DL.  To  echo  this  point,  none  of  the  50  military  DL  studies  that  ESC  CAFC2  obtained 
via  the  Defense  Technical  Information  Center  (DTIC)  between  1995-2000  included  empirical  data  on  the 
effectiveness  of  DL  in  a  web-based  environment.  This  study  will  be  one  of  the  first  to  track  USAF  dis¬ 
tributed  learning  effectiveness  in  a  C2  environment. 
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4.0— TBMCS  TRAINING  CONCEPT,  DESIGN  AND  DEVELOPMENT 

Kemp,  Morrison,  Ross  (1998)  suggested  before  starting  an  instructional  development  project,  man¬ 
agement  should  ask  themselves  “Why  do  we  need  instruction?”  For  TBMCS,  it  was  evident  that  users 
needed  to  learn  how  to  use  the  software  application  when  it  arrived  at  their  desktop.  Specific  job  training 
has  precise,  immediate  requirements  with  identifiable  and  often  measurable  outcomes.  The  training  mate¬ 
rial  development  must  stress  the  teaching  of  knowledge  and  skills  for  the  performance  of  assigned  tasks. 
Kemp,  Morrison,  &  Ross  (1998),  US  Air  Force  Instmction  36-22  (1997),  and  Clark  (2000)  all  identify  a 
common  ISD  process  consisting  of  the  following: 

•  Analysis — ^identifying  tasks  and  skills  requiring  training 

•  Design — ^identifying  the  objectives,  test  questions,  and  sequencing  of  instmction 

•  Development — creating  the  courseware  and  activities 

•  Implementation — conducting  the  training 

•  Evaluation — ^reviewing  the  training  design,  development,  and  implementation  of  the  course 

Since  the  ISD  process  provides  a  stmctured,  systematic  means  of  providing  training,  the  TBMCS 

training  development  contractor  was  required  to  follow  the  process  to  complete  the  distributed  training 
effort.  The  following  best  practices  and  lessons  learned  are  identified  from  each  phase  of  the  ISD  process: 

4.1  REQUIREMENTS  IDENTIFICATION-THE  ANALYSIS  PHASE 

Due  to  cost  constraints  at  ESC,  the  developing  contractor  was  not  requested  to  conduct  a 
Task  and  Skill  Assessment  (TASA)  to  determine  the  precise  tasks  and  skills  required  for  training 
on  TBMCS.  Instead,  the  key  tasks  identified  as  training  requirements  from  the  legacy  TASAs 
were  deemed  adequate  to  transfer  to  the  TBMCS  training  development  contract.  Jonassen,  Han- 
num  and  Tessmer  (1989)  stated  that  the  TASA  is  probably  the  most  important  component  of  the 
ISD  process.  All  future  instructional  strategies  and  decisions  are  based  upon  the  results  of  the 
TASA.  Thus,  the  quality  of  the  task  analysis  determines  the  quality  of  instruction.  Wolfe  et  al. 
(1991)  stated  that  once  job  tasks  are  identified  they  must  be  prioritized  to  focus  on  key  tasks  that 
are  most  critical,  difficult,  and  frequently  used.  Without  an  original  TBMCS  TASA,  the  devel¬ 
oping  contractor  was  required  to  provide  training  materials  supporting  all  2,140  tasks  identified 
from  the  legacy  TASAs.  This  decision  is  critical  in  understanding  the  direction  of  the  training 
contract  from  its  conception.  This  proved  to  be  a  poor  decision  that  resulted  in  additional  costs  to 
the  training  contract. 
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4.2  THE  COST  OF  PAPER-THE  DESIGN  PHASE 

DOD  I  5000.2  states  that  training  for  major  weapon  system  components  shall  not  be  procured  before 
the  weapon  system  hardware  and  software  design  stabilizes.  However,  the  software  acquisition  life  cycle 
identified  in  API  63-123  requires  prototypes,  tests,  and  low  rate  initial  production  of  C2  applications 
during  a  6-18  month  spiral  development  schedule.  Each  TBMCS  spiral  test  requires  hundreds  of  testers  to 
be  trained.  As  a  result,  training  materials  must  be  developed  simultaneously  with  software  development. 
This  poses  a  significant  problem  for  the  training  developer.  Each  time  the  software  is  modified  there  is  an 
equal  and  parallel  effort  required  in  updating  and  distributing  the  training  materials.  Paper-based  training 
materials  for  a  large  C2  system  are  costly.  During  the  time  of  contract  award  in  1995  the  estimated 
number  of  users  (operators,  system  and  network  administrators)  was  approximately  2700.  It  was 
assumed  at  this  time  that  all  students  would  receive  copies  of  all  training  materials  (student  mate¬ 
rials,  programs  of  instruction,  and  lesson  plans).  Figure  4.2-1  depicts  the  cost  to  produce  one 
hard  copy  of  the  full  set  of  TBMCS  training  materials 


Cost  of  Paper  Based  Training  Materials 

Paper-based  T raining  Materiais  for  Operators, 

System  Administrators  and  Network  Administrators 

1 07  modules  x  500  pages  x.07  per  page 
for  reproduction  *  2700  students 

$10,111,500.00 

Paper-based  Program  of  Instructions  (POIs)  for  Op¬ 
erators, 

System  Administrators  and  Network  Administrators 

5  POIs  X  60  pages  x.07  per  page 
for  reproduction  *  2700  students 

$5,670,000.00 

Paper  Based  Lesson  Pians  (LPs)  for  Operators, 

System  Administrators  and  Network  Administrators 

5  LP  X  60  pages  x.07  per  page 
for  reproduction  *  2700  students 

$1,890,000.00 

Total  Paper-based  Training  Efforts 

$17,671,500.00 

Figure  4.2-1.  The  Cost  of  Paper 

When  framing  requirements  are  identified  early  in  a  program  life  cycle,  development  costs 
can  be  budgeted  in  incremental  amounts  over  the  life  of  the  contract.  The  original  cost  estimate 
of  TBMCS  training  material  development  and  implementation  was  expected  to  be  no  greater 
than  10%  of  the  overall  software  development  effort.  TBMCS  training  development  costs  upon 
contract  conception  in  1995  were  anticipated  to  be  $40  million  dollars  that  were  to  be  divided 
into  progressive  increments  over  the  five-year  contract.  With  an  estimated  $17  million  in  repro¬ 
duction  costs  alone  by  1997,  ESC  quickly  realized  that  a  major  change  in  software  functionality 
requiring  updates  and  dissemination  the  training  materials  would  seriously  affect  the  TBMCS 
training  development  budget.  Campbell  and  Bourne  (1997)  stated  that  at  some  point  educators 
and  trainers  will  use  the  web  to  reduce  costs,  increase  quality  and  increase  the  rate  of  new 
knowledge  and  innovation  about  how  to  support  learning — or  disappear  themselves.  With  the 
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high  cost  of  reproduction,  it  appeared  the  web — a  technology  that  enables  students  to  access 
materials  in  a  timely  manner  without  distribution  costs — was  the  delivery  mechanism  of  choice 
for  TBMCS.  By  mid  1999,  the  TBMCS  material  conversion  from  MS  Word  to  HTML  was  com¬ 
pleted.  Customers  of  this  distributed  training  were  anticipated  to  be  5,000  multi-service  operators 
and  system  administrators  who  use  TBMCS  in  their  wartime  duties.  The  goal  was  justifiable — 
users  had  immediate  access  to  training  material  anytime,  anywhere,  and  anyplace  as  required  by 
Joint  Vision  2010  and  the  DOD  strategic  plan/ADL  initiative. 

4.3  THE  TECHNOLOGY  CHASE-THE  DEVELOPMENT  PHASE 

In  theory,  web-based  training  provides  immediate  access  to  training  materials  as  the  acquisition  life 
cycle  progresses,  and  the  software  is  updated.  Although  the  materials  were  completed  and  accessible  to 
multi-service  personnel  anytime,  anywhere,  and  anyplace,  there  were  still  unresolved  problems.  In  devel¬ 
oping  TBMCS  training  materials,  the  “knee  jerk”  reaction  of  converting  to  HTML  as  a  media  selection 
without  a  technology  assessment  of  supporting  architecture  was  a  serious  problem,  which  ultimately  led 
to  the  cancellation  of  the  web  site.  Ely  (2000)  stated  that  the  msh  to  jump  onto  the  distance  education 
bandwagon  is  understandable  in  light  of  several  factors:  (1)  everyone  is  doing  it,  (2)  the  promise  of  in¬ 
come  and/or  savings,  (3)  the  ubiquitous  presence  of  computers  and  networks  available  for  users,  and  (4) 
the  number  of  users  who  are  not  being  reached  by  conventional  education.  Conversely,  McNabb  (2000) 
stated  that  historically  implementing  technology  for  technology’s  sake  without  regard  for  how  the  use  of 
the  technology  will  be  integrated  with  the  curriculum  has  failed.  She  further  stated  that  a  lesson  learned 
from  past  technology  implementation  efforts  is  that  a  technology  needs  assessment  is  as  critical  as  a 
TASA. 

Somewhere  in  the  quest  for  technology  and  the  desire  to  “hop  on  the  web  bandwagon”,  the  TBMCS 
technology/infrastmcture  plan  was  forgotten.  As  users  from  worldwide  locations  attempted  to  access  the 
materials,  three  problems  became  apparent.  The  first  was  accessing  materials  from  remote  locations,  the 
second  problem  was  “bandwidth”  (the  real  end-to-end  communications  speed  for  users),  and  the  third  was 
local  computer  security  initiatives  hindering  the  use  of  the  web  server  at  user  locations.  These  issues  sig¬ 
nificantly  affected  the  success  of  the  distributed  learning  initiative.  The  number  of  server  hops  required  by 
users  in  locations  such  as  Korea,  Hawaii,  and  Germany  to  access  the  Colorado  server  were  so  numerous 
that  most  attempts  were  timed  out,  students  became  frustrated  and  often  quit  before  completing  the  train¬ 
ing.  To  resolve  the  problem  three  additional  servers  were  placed  at  worldwide  locations  to  lessen  the  geo¬ 
graphic  distance  between  the  clients  and  servers.  This  effort  was  completed  in  August  2000.  The  cost  of 
this  effort  is  shown  in  Figure  4.3-1. 
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Training  infrastructure  for  servers 


HW  For  Servers 


Distributed  Servers 

22,835.11 

9 

$205,516.00 

Duplex  Ultra  2  Chasis  Upgrade 

$427.00 

Hot  Plug  Power  Supply 

$943.00 

Prolleant  6000  Internal  Drive  Gage,  SCSI-3 

$298.00 

Ram  up-grades 

$3,591.00 

Redundant  Fan  Kit 

$309.00 

Smart  Array  3200  Controller 

$1,737.00 

Laptops  for  MTTs 

3,541.05 

20 

$70,821.00 

Total  HW  for  Severs 

$283,642.00 

SW  For  Servers 

Microsoft  Windows  NT  4.0 

$15,577.00 

server  doe  kit,  server  license. 

service  pack  CD 

McAfee  Anti-Virus  v4.03 

no  cost 

Netscape  Navigator  v4. 07 

no  cost 

Netscape  security  certificates 

8 

$2,000.00 

Oracle  Enterprise  Edition  v8l 

12,444.00 

9 

$111,996.00 

Plateau  Enterprise  v3.1 

9 

$172,750.00 

WebTrends  Professional  Suite  v4.0 

8 

$7,664.00 

MIsc  Server  SW 

$1,023.00 

Total  SW  for  Servers 

$309,987.00 

Total  HW/SW  for  Servers 

$821,368.00 

Figure  4.3-1.  Infrastructure  Costs 

Bandwidth  (response  times  for  end  users  over  the  available  communications  path)  was  a  great  con¬ 
cern  on  the  military  bases  containing  the  newly  placed  distributed  servers.  During  an  independent  assess¬ 
ment  conducted  January  through  March  2001  by  MITRE  Corporation,  with  users  initiating  access  via 
private  internet  service  providers,  access  times  to  the  training  server  at  Hurlburt  AFB  in  Florida  were 
more  than  three  times  as  slow  as  access  times  to  the  contractor’s  server  in  CO.  In  another  specific  test, 
simple  “ping”  commands  were  transmitted  between  the  Colorado  Springs,  CO  facility’s  training  server 
and  the  corresponding  server  at  Hurlburt  AFB  FF  to  assess  the  round  trip  communication  time  over  the 
NIPRNET  network.  Round  trip  times  were  repeatedly  documented  as  taking  10  seconds.  Conversely, 
when  simple  “pings”  were  transmitted  between  civilian  locations  (MITRE’ s  Bedford,  MA  facility  through 
numerous  routers,  over  a  dedicated  T3  communications  line,  to  a  server  at  the  MITRE  Washington,  DC 
office)  round  trip  times  were  10  milliseconds.  Plateau  required  Oracle  replication  of  the  student  databases 
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between  locations.  However,  with  the  military  facility  NIPRNET  infrastructure  taking  1,000  times  as  long 
to  accept  and  return  simple  ping  transmissions,  the  feasibility  of  expanding  database  synchronization  to  a 
worldwide  set  of  servers  was  judged  to  be  very  doubtful. 

Student  database  transmission  also  requires  ports  in  the  firewalls  to  be  accessible  to  exchange  data. 
Local  military  base  fhewall  policies  hindered  successful  operation  of  the  distributed  servers.  Security 
managers  at  certain  locations  would  not  allow  Oracle  database  exchanges  to  occur  through  theh  firewall. 
Without  an  approved  certification  and  accreditation  package,  distributed  servers  could  not  be  used  for 
training. 

4.4  SUMMATIVE  REVIEW  OF  THE  IMPLEMENTATION— THE  EVALUATION  PHASE 

The  TBMCS  C2  software  system  and  associated  web-based  training  was  fielded  October  2000 
through  June  2001  based  upon  System  of  Record  (SOR)  decision  by  the  Joint  Configuration  Control 
Board  (JCCB).  Mobile  Training  Teams  (MTTs)  traveled  to  force  level  locations  to  facilitate  the 
distributed  training  for  the  various  system  administrators,  operators,  and  Perimeter  Security  System  (PSS) 
network  administrators.  A  preliminary  evaluation  was  conducted  October-December  2000  at  the  first  four 
fielded  locations.  As  users  began  to  access  the  training  materials,  student  feedback  was  negative  due  to 
scrolling  text  pages  and  lack  of  interactivity.  This  design  defied  many  aspects  of  learning  theory  and  web 
design.  Nielson  (1997)  identified  long  scrolling  pages  as  the  sixth  most  common  mistake  in  web  design. 

He  stated  only  10%  of  users  scroll  beyond  the  information  that  is  visible  on  the  screen  when  a  page  comes 
up.  All  critical  content  and  navigation  options  should  be  on  the  top  part  of  the  page.  Moore  (1989)  defined 
interactivity  in  distance  learning  as:  a)  between  the  learner  and  the  content;  b)  between  the  learner  and  the 
instmctor;  and  c)  between  the  learner  and  peers.  TBMCS  distributed  training  lacked  all  levels  of 
interactivity.  Results  of  the  student  End  of  Course  (EOC)  critiques,  student  achievement  tests  and  focus 
groups  are  shown  in  Eigure  4.4-1: 
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Location 

Legacy  Ex¬ 
perience 

TBMCS 

Experience 

Satisfaction 

Means 

Average 

Pre  Test 
Scores  Be¬ 
fore  Train¬ 
ing 

Average 
Post  Test 
Scores  Af¬ 
ter  Training 

Focus  Group  Com¬ 
ments 

Shaw 

50% 

60% 

74%  overall 

63%  materi¬ 
als 

61% 

86% 

Training  lacked  inter¬ 
activity.  Facilitators 
lacked  operational 
knowledge  ef  the 
system. 

Osan 

75% 

89% 

Training  lacked  inter¬ 
activity.  Facilitators 
lacked  operational 
knowledge  ef  the 
system. 

Miramar 

0% 

23% 

80%  overall 

75%  material 

52% 

88% 

Training  lacked  inter¬ 
activity.  Facilitators 
lacked  operational 
knowledge  ef  the 
system. 

Elmendorf 

38% 

0% 

72%  overall 

65%  material 

58% 

89% 

Training  lacked  inter¬ 
activity.  Facilitators 
lacked  operational 
knowledge  ef  the 
system. 

Figure  4.4-1.  Preliminary  Feedback  Distributed  Learning 

Although  the  data  was  preliminary,  the  EOC  critiques  suggest  that  the  students  were  not  “satisfied” 
with  the  concept  of  self-paced  distributed  learning  with  little  human  interaction;  however,  pre-/post-test 
scores  revealed  that  students  understood  facts  and  concepts,  which  suggests  that  knowledge  “achieve¬ 
ment”  resulted  from  the  training.  To  better  understand  why  the  user  satisfaction  was  low,  ESC  focus 
groups  were  conducted  with  the  students  at  the  training  events.  The  largest  single  improvement  the  users 
wanted  to  see  was  facilitator-led  interactivity  with  the  “TBMCS  system”  versus  the  “TBMCS  training 
materials”.  Students  believed  training  would  be  greatly  enhanced  if  they  had  facilitator-led,  hands-on  ex¬ 
ercises  versus  facilitating  web-based  training.  User  perception  weighs  heavily  on  ESC  training  acquisition 
strategy.  Although  distributed  learning  environments  are  dictated  by  DOD  policy,  user  preferences  are  for 
MTTs. 

This  requirement  was  identified  to  the  TPT  for  validation.  ESC  was  requested  to  a)  enhance  the  web- 
based  training  by  adding  interactivity,  and  b)  shift  from  a  distributed  learning  environment  back  to  a  tra¬ 
ditional  instmctor-led,  “hands-on”  approach  to  training.  The  type  of  interaction  chosen  was  between  the 
“learner  and  the  content”.  Roblyer  and  Ekhaml  (2000)  defined  this  type  of  interaction  as  “reciprocal 
events  requiring  two  objects  and  two  actions”.  They  identified  an  instructional  goal  of  interaction  as  en¬ 
couraging  reflection  and/or  discussion  on  course  topics  and  concepts  by  utilizing  instmctional  design  to 
increase  the  participation  and  feedback.  The  tool  of  choice  to  implement  interactivity  for  the  TBMCS 
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training  materials  was  Macromedia’s  Dreamweaver.  Vora  (1998)  stated  What  You  See  is  What  You  Get 
(WYSIWYG)  editors  function  like  word-processing  or  desktop  publishing  programs,  allowing  authors  to 
lay  out  pages  as  they  want,  and  the  WYSIWYG  editors  write  the  necessary  HTML  code  in  the  back¬ 
ground.  The  updated  training  materials  were  posted  on  the  web  server  in  June  2000.  Technology  inser¬ 
tion/conversion  efforts  include  new  hardware,  software  and  training.  Cost  for  the  Hardware/Software 
(HW/SW)  material  updates  are  shown  in  Figure  4.4-2. 


Cost  Per  Unit 

Number  of  Units 

Total 

Development  &  Integration 

HW  For  46  training  developers 

2,065.00 

46 

$94,990.00 

SW  For  training  developers 

Web  Trends 

499.00 

46 

$22,954.00 

Astra  Site  Manager 

499.00 

46 

$22,954.00 

Dreamweaver  HTML  Editor  & 
Course  Buiider 

722.00 

46 

$33,212.00 

Repiacement  SW  from  Unix  to  PC 

$25,168.00 

Paint  Shop 

46.00 

46 

$2,116.00 

Total  SW  For  Servers 

$106,404.00 

Total  Development  &  Integration  costs  for  infrastructure  $201,394.00 


Figure  4.4-2.  Cost  to  Upgrade  HW/SW  and  Skill  Training 

Facilitator-led  Practical  Exercises  (PEs)  were  added  to  the  course  curriculum  to  fulfill  the  require¬ 
ment  for  instructor-led,  hands  on.  Instructors  who  participated  in  the  earlier  fielding  gained  valuable  h- 
sight  into  the  user  requirements.  An  important  discovery  was  that  not  all  operators  (operations,  plans, 
intelligence)  required  training  on  all  products.  PEs  were  designed  to  focus  on  duty  positions  and  job  tasks 
within  the  AOC.  This  allowed  a  clearer  division  to  be  made  between  the  training  provided  within  the 
AOC  cells.  The  tasks  and  skills  that  were  used  most  frequently  were  identified  as  candidates  for  facilita¬ 
tor-led  PEs. 

4.5  TBMCS  1.0.1— FIELDING  AND  TRAINING  PROCESS 

The  primary  objective  of  TBMCS  training  is  to  attain  and  maintain  the  capability  to  operate  and  ad¬ 
minister  the  system.  A  secondary  objective  is  to  develop  advanced  skills  that  facilitate  increased  effec¬ 
tiveness  of  the  system.  These  objectives  are  met  through  type-1  training.  AEI  36-2201  identifies  type-1 
training  as  “contract  training”  or  “factory  training”  that  AETC  arranges  for  Air  Eorce  and  other  DOD  per¬ 
sonnel  and  contractors  to  conduct  at  either  the  contractor’s  location  or  a  DOD  facility.  Due  to  large  num¬ 
bers  of  geographically  dispersed  personnel  requiring  TBMCS  training,  surge  training  of  100  percent  of 
the  TBMCS  user  population  was  not  economically  or  physically  possible.  Thus,  a  train-the-trainer  phi¬ 
losophy  was  chosen  and  approved  by  the  Joint  Air  Operations  (JAO)  Training  Planning  Team  (JAOTPT). 
Initial  train-the-trainer  training  for  TBMCS  1 .0. 1  was  provided  for  personnel  with  previous  legacy  system 
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experience.  In  theory,  this  approach  provides  training  to  a  core  cadre  of  personnel  from  all  locations, 
which  then  relied  on  those  students  to  train  remaining  unit  personnel  through  On-the-Job-Training  (OJT). 

Limited  initial  cadre  training  was  provided  to  operators  and  system  administrators  via  MTTs  at  se¬ 
lected  regional  sites  worldwide  based  on  the  train-the-trainer  concept.  Training  was  targeted  for  experi¬ 
enced  legacy  operators  and  system  administrators.  The  degree  of  training  was  constrained  to  differences 
between  the  SOR  of  the  legacy  systems  and  TBMCS  1.0.1.  A  fielding  decision  -i-  300  days  was  antici¬ 
pated  for  the  services  to  complete  installation,  training,  system  accreditation,  OJT,  and  system  cutover. 
Type-1  training  for  system  administrators  and  operators  began  at  selected  locations  30  days  after  the  SOR 
decision.  System  administrators  were  trained  on  TBMCS  installation  processes  by  means  of  loading  and 
configuring  a  training  suite.  Upon  successful  build  of  the  training  suite,  operators  were  then  provided 
hands-on  training  on  the  training  suite.  Type-1  training  also  included  PSS  installation  and  training  for 
network  administrators,  and  exercise  support  for  operators.  Training  was  fielded  as  shown  in  Figure  4.5-1. 
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Duration  of  Event 

Event 

Comment 

0 

- 

Fielding  Decision 

30 

1 00  day  total 

SW  Kit  distribution 

This  is  only  first  SW  shipment 

30 

1 80  day  total 

Type-1  Training  (Build  Training 
Suites  and  Conduct  Initial  Op¬ 
erator  Training 

Sys  Ad  and  Operator  to  be  scheduled 
sequentially 

Sys  Ad 

todays  easite 

Operator 

5  days  ea  site 

PSS 

3-5  days  ea  site 

100 

- 

S/W  delivered  to  all  locations 

SW  Kits  completed 

210 

- 

Type-1  Training  Completion 

210 

Units  are  given  90 
days  as  part  of  fielding 
to  complete  OJT 

Begin  OJT 

300 

- 

Complete  OJT 

300 

- 

System  Ready  for  Cutover 

Figure  4.5-1.  Training  Process 


TBMCS  System  Administrator  Training — consisted  of  two  weeks  of  lectures  and  hands-on  installation 
training  facilitated  by  two  MTT  SysAds.  As  part  of  the  training,  MTTs  provided  guidance  as  the  SysAds 
loaded  the  TBMCS  software.  On  completion  of  training,  trainees  are  expected  to  assist  in  establishing  a 
training  program  to  train  remaining  unit  SysAds.  All  users  will  maintain  proficiency  through  OJT  or  other 
subsequent  training. 

TBMCS  Force  Level  Operator  Training — consisted  of  five  days  of  MTT  facilitator-led  practical  exer¬ 
cises  augmented  by  web-based  training.  MTTs  consisted  of  one  plans,  one  operations,  and  one  intelli¬ 
gence  trainers  at  regional  locations.  During  this  time  services  were  encouraged  to  utilize  local  Subject 
Matter  Experts  (SMEs)  to  assist  with  the  training.  Due  to  the  complexity  of  the  TBMCS  system  it  was 
advantageous  to  have  SMEs  knowledgeable  in  local  doctrine/mission  orientation  to  assist  personnel  being 
trained. 
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This  training  focused  on  the  functionality  required  to  perform  joint  air  operations  in  the  AOC  and 
consisted  of  three  separate  tracks.  Service  training  representatives  separated  students  into  operations, 
plans,  and  intelligence  tracks.  The  tracks  focused  on  a  job  specific/tasked  based  approach  as  follows: 
Force  Level  Combat  Operations — Specific  positions  within  the  operations  cell  were  grouped  together 
into  the  following  training  categories  that  focused  the  training  on  applications  and  concepts  of  employ¬ 
ment  required  to  perform  their  positional  duties  in  an  AOC.  Force-level  combat  operations  training  tasks 
were  mirrored  with  their  respective  TBMCS  application  with  supporting  operations  checklists  to  assist 
operators. 

•  Director/Chief  Combat  Operations  (DCO/CCO),  Deputy  CCO,  and  Senior  Offensive  Duty  Offi¬ 
cer  (SODO) 

•  Offensive  Duty  Officer 

•  Time  Sensitive  Targeting/Time  Critical  Targeting 

•  Defensive  Duty  Officer 

•  Air  Tasking  Order  Replanning 

•  Weather 

•  Reports 

•  Airspace 

Force  Level  Combat  Plans — Specific  positions  within  the  plans  cell  were  grouped  together  into  the  fol¬ 
lowing  training  categories  that  focused  the  training  on  applications  and  concepts  of  employment  required 
to  perform  their  positional  duties  in  an  AOC.  Force-level  combat  plans  training  tasks  were  mirrored  with 
their  respective  TBMCS  application  with  supporting  plans  checklists  to  assist  operators. 

•  Chief  Combat  Plans/Deputy  Chief  Combat  Plans 

•  Air  Tasking  Order  Production 

•  Air  Tasking  Order  Planner 

•  Air  Tasking  Order  MAAP  Development 

•  Airlift 

•  Airspace 

Force  Level  Combat  Intelligence — Specific  positions  within  the  intelligence  cell  were  grouped  together 
into  the  following  training  categories  that  focused  the  training  on  applications  and  concepts  of  employ¬ 
ment  required  to  perform  their  positional  duties  in  an  AOC.  Force-level  combat  intelligence  training  tasks 
have  been  mirrored  with  their  respective  TBMCS  application  with  supporting  intelligence  checklists. 

•  Analysis  Cell 

•  Operations  Intelligence 
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•  Imagery  Intelligence 

•  ELINT/Analyst 

•  Data  Base  Manager 

•  Plans  Intelligence 

•  Combat  Assessment 

•  Targeteer/Plans 

•  ATO  Execution  Intelligence 

•  ELINT/ATO  Execution 

•  Targeteer/ATO  Execution 

•  Analyst/ATO  Execution 

Perimeter  Security  System  Training — PSS  training  consisted  of  two  weeks  of  SW  training  and  installa¬ 
tion  of  the  PSS  HW.  PSS  is  the  security  enclave  embedded  in  TBMCS.  MTTs  assisted  the  network  ad¬ 
ministrators  in  installing  the  PSS  HW,  and  provided  briefings  on  the  individual  components  (external 
routers,  virtual  private  network,  safenet  enterprise  management  components,  and  internal  routers)  that 
comprise  the  PSS  for  TBMCS. 


20 


Training  Program  Review 
TBMCS  Training 
Program  Evaluation  v.  1.0.1 

5.0— DATA  GATHERING 

The  purpose  of  this  section  is  to  describe  the: 

•  Research  questions  answered  in  this  study 

•  Population  of  this  study 

•  Evaluation  model  used  in  this  study 

•  Instruments  used  to  collect  data  relevant  to  the  study 

•  Procedures  used  to  collect  the  data 

Due  to  the  numerous  training  baseline  changes  conducted  during  the  overall  software  development 
evolution  and  the  inconsistencies  of  data  collection  this  study  did  not  lend  itself  to  a  hypothesis  testing 
approach.  Instead,  an  exploratory  research  methodology  was  chosen  to  support  the  research  questions . 

5.1  RESEARCH  QUESTIONS 

•  Research  Question  1 :  Were  the  majority  of  students  satisfied  at  the  completion  of  training? 

•  Research  Question  2:  Will  there  be  a  difference  in  the  students  test  scores  after  completing  the 
training? 

•  Research  Question  3:  Will  users  be  confident  in  their  ability  to  perform  key  tasks  upon  compb- 
tion  of  the  training? 

•  Research  Question  4:  Is  there  a  correlation  between  user  experience  and  EOC  satisfaction? 

•  Research  Question  5:  Will  the  students  perceive  the  facilitator  as  knowledgeable  about  the 
course  content? 

•  Research  Question  6:  Will  students  perceive  that  the  course  covered  the  key  TBMCS  skills  spe¬ 
cific  to  their  work  center? 

•  Research  Question  7:  Will  students  perceive  that  their  units  provided  a  workspace  that  sup¬ 
ported  a  successful  training  environment? 

5.2  STUDENT  POPULATION 

Upon  system  fielding,  the  total  TBMCS  user  population  is  anticipated  to  be  5,000  multi-service  system 
administrators,  operators  and  network  administrators.  The  train-the-trainer  methodology  trained  a  limited  cadre 
of  approximately  800  with  MTTs  at  21  locations.  The  trainees  are  geographically  dispersed  throughout  multi¬ 
ple  locations  in  the  Continental  United  States  (CONUS),  and  Pacific  and  European  countries.  API  131-AOC, 
Volume  3,  identifies  the  duty  positions  associated  with  the  force-level  operation  of  an  air  operations  system. 
TBMCS  operators,  system  administrators,  and  network  administrators  include  contractors,  military  enlisted 
personnel,  and  officers.  Students  are  both  female  and  male,  and  range  in  age  from  20-45  years  with  various 
educational  backgrounds  and  experience  levels.  Training  was  conducted  at  the  students  home  station. 
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A  force-level  operator  course,  a  system  administrator  course  and  a  PSS  course  were  taught  at  21  locations.  The 
total  Trained  Personnel  Requirement  (TPR)  is  better  understood  based  upon  focus  group  discussions  and  Sys¬ 
tem  Program  Office  (SPO)  observation  during  fielding.  TPR  for  total  joint  operators  is  approximately  1350, 
joint  system  administrators  is  300,  and  joint  network  administrators  is  100. 

5.3  DATA  COLLECTION  MODEL 

The  reason  for  evaluation  is  to  determine  the  effectiveness  of  a  training  program.  When  the  evalua¬ 
tion  is  done,  we  can  hope  that  the  results  are  positive  and  gratifying,  both  for  those  responsible  for  the 
program  and  for  upper-level  managers  who  will  make  timely  decisions  based  on  evaluation  results.  To 
demonstrate  the  effectiveness  and  value  of  the  TBMCS  distributed  training  program,  Kirkpatrick’s  theory 
of  evaluation  was  used.  The  following  theories  were  evaluated  prior  to  selecting  a  model: 

•  Kirkpatrick’s  Theory  of  Evaluation 

•  Stufflebeam’  CIPP  model 

•  Stakes  Countenance  Model 

•  Sanders  and  Nifziger  Checklist 

•  Gowin  and  Millman  QUEMAC  model 

•  Worthen  and  Sanders  -  Scriven’s  MEC  (meta-evaluation  checklist) 

Kirkpatrick’s  model  was  designed  for  practitioners  in  the  training  field  who  plan,  implement,  and 
evaluate  training  programs.  It  was  primarily  chosen  over  the  other  models  due  to  high  usage  rates,  and 
validity  for  use  by  industry  and  Government.  Eigure  5.3-1  shows  the  Kirkpatrick  IV  Levels  of  Evaluation. 


Level 

Evaluation 

Explanation 

TBMCS  Data  Gathering 

1 

Reaction 

Assesses  participants’  initiai  reac¬ 
tions  to  a  course.  This  in-turn, 
offers  insights  into  participants 
satisfaction  with  a  course,  a  per¬ 
ception  of  vaiue. 

A  questionnaire  was  used  to  gather 
quantitative  data.  A  focus  group  was 
conducted  to  gather  qualitative  data 

II 

Learning 

Assesses  the  amount  of  informa¬ 
tion  that  participants  iearned. 

A  knowledge-based  pre-  and  post¬ 
test  was  used  to  assess  the  amount 
of  information  learned. 

III 

Transfer 

Assesses  the  amount  of  materiai 
that  participants  actuaiiy  use  in 
everyday  work  after  taking  the 
course. 

Students  were  asked  to  rate  their 
ability  to  perform  key  tasks  after 
training 

IV 

Business  Resuits 

Assesses  the  financiai  impact  of 
the  training  course  on  the  bottom 
iine  of  the  organization  six  months 
to  two  years  after  course  compie- 
tion. 

Collecting  data  to  identify  experi¬ 
ence  levels,  turnover  rates,  chang¬ 
ing  experience  levels  during  test, 
and  operational  readiness  inspection 
results  is  a  longitudinal  study  not 
included  in  this  report. 

Figure  5.3-1.  Kirkpatrick  IV  Levels  of  Evaluation 
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5.4  INSTRUMENTATION 

Overall  data  was  gathered  using  four  collection  methods: 

•  Surveys 

•  Focus  Groups 

•  Pre-/Post-tests 

•  Student  Self  Assessment 

5.4.1  Survey 

Instructors  were  tasked  to  prepare  an  EOC  survey  to  collect  data  on  the  effectiveness  of  the  train¬ 
ing  program.  This  survey  can  be  found  in  Appendix  1  of  this  document.  The  specific  objectives  of  the  survey 
were  to  obtain: 

•  valuable  feedback  to  help  evaluate  the  program 

•  comments  and  suggestions  for  improving  the  program 

•  quantitative  information  that  can  be  used  to  establish  standards  of  performance  for  future  pro¬ 
grams  as  explained  in  Kirkpatrick’s  Level  I  Evaluation-Reaction 

•  quantitative  feedback  to  be  used  with  the  survey  to  validate  user  satisfaction  as  explained  in 
Kirkpatrick’s  Level  I  Evaluation-Reaction 

5.4. 1.1  Data  Gathering  Procedures  of  the  Survey 

At  the  beginning  of  training,  MTT  facilitators  requested  students  to  annotate  their  reactions  to  train¬ 
ing  on  an  EOC  critique.  The  students  were  informed  of  the  location  of  the  critique  and  encouraged  to 
document  their  comments  throughout  the  duration  of  the  course.  Instructors  informed  students  that  their 
input  provides  feedback  on  the  effectiveness  of  the  course  and  their  comments/suggestions  help  to  plan 
future  courses  to  meet  the  students’  needs  and  interests.  At  the  end  of  the  course,  MTT  facilitators  again 
informed  students  of  their  obligation  to  provide  feedback  as  to  the  effectiveness  of  the  TBMCS  training. 

5.4.2  Focus  Group 

Eocus  groups  are  moderated  group  discussions  designed  to  encourage  free-flowing  disclosures  be¬ 
tween  students.  TBMCS  focus  groups  included  ESC  training  representatives  and  students.  Eocus  groups 
collect  qualitative  data  and  offer  rich  insights  into  the  subject  matter.  Group  dynamics  and  shared  ideas 
provide  results  not  obtainable  from  other  research  methods. 

Specific  objectives  of  this  focus  group  were  to: 

•  Obtain  qualitative  feedback  to  be  used  with  the  Survey  to  validate  user  satisfaction  as  explained 
in  Kirkpatrick’s  Level  I  Evaluation-Reaction 

•  Identify  a)  user  expectations,  b)  the  satisfaction  level,  c)  problems  occurred,  and  d)  areas  for 
improvement. 
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5.4.2. 1  Data  Gathering  Procedures  of  the  Focus  Group 

1.  The  TBMCS  training  representative  met  with  the  instructors  whose  classes  where  selected  for 
the  study  to  introduce  the  project  as  well  as  to  inform  the  instmctor  what  questions  were  to  be 
asked  during  the  focus  group. 

2.  TBMCS  training  was  given  to  all  students,  thus  the  focus  group  included  all  students  partici¬ 
pating  in  the  course.  MTT  facilitators  were  asked  to  leave  the  room. 

3.  The  TBMCS  training  representative  met  with  the  students  and  asked:  a)  what  their  barning  ex¬ 
pectations  were,  b)  to  provide  feedback  on  the  course,  and  c)  to  provide  areas  for  improvement 
(if  any)  for  follow-on  courses.  These  questions  were  open  ended  to  allow  for  student  collabora¬ 
tion  amongst  the  group. 

5.4.3  Pre  -test 

Standardized  tests  are  designed  to  fairly  measure  student  achievement  in  different  academic  subjects. 

TBMCS  test  questions  supporting  training  objectives  were  originally  identified  in  the  design  phase  as  the 

TBMCS  training  material  was  developed. 

The  specific  objectives  of  the  pre-test  were  to: 

•  obtain  initial  data  to  compare  with  the  post-test  to  validate  the  transfer  of  knowledge  as  ec- 
plained  in  Kirkpatricks  Level  II  Evaluation-Learning 

•  Help  instmctors  determine  the  strengths  and  needs  of  students  in  order  to  work  with  them  to 
improve  their  individual  academic  skills 

•  Provide  information  to  instmctional  designers  to  help  determine  how  well  training  assisted  users 
in  learning 

5.4.3. 1  Data  Gathering  Procedures  of  the  Pre-Test 

1.  ESC  met  with  the  instmctors  whose  classes  were  selected  for  the  study  to  introduce  the  project 
as  well  as  to  verify  how  the  instmctor  will  administer  the  instmment. 

2.  MTT  facilitators  administered  computer-generated  pre-tests  to  all  the  students  participating  in 
the  course. 

3.  Answers  to  pre-test  questions  were  collected  electronically  for  each  student  involved  in  TBMCS 
training. 

5.4.4  Post-test 

At  the  completion  of  training,  the  pre-test,  administered  prior  to  the  training,  was  re-administered  as 

a  post- test  to  all  students  to  determine  if  the  students’  knowledge  had  improved. 
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The  specific  objectives  of  the  survey  were  to: 

•  Correlate  pre-  and  post-test  scores  to  validate  if  a  learning  transfer  took  place  as  explained  in 
Kirkpatrick’s  Level  III-Leaming 

5.4.4. 1  Data  Gathering  Procedures  of  the  Post-test 

The  same  procedure  followed  during  the  pre-test  was  followed  to  administer  the  post-test. 

5.4.5  Self-Assessment 

The  self-assessment  allows  the  instmctors  to  gain  an  awareness  of  the  confidence  a  student  has  in 
their  ability  to  complete  key  tasks.  It  is  not  necessarily  an  accurate  evaluation  of  an  individual’s  ability, 
but  does  indicate  how  confident  the  training  has  left  them.  It  is  an  indirect  indication  of  satisfaction  with 
the  training  and  how  well  they  learned  what  was  taught .  A  likert  scale  of  1-5  was  utilized.  l=Can’t  per¬ 
form;  2=Perform  with  over-the-shoulder  assistance;  3=Perform  with  only  on-line  help;  4=Perform  with¬ 
out  assistance;  5=Did  not  attempt  task.  Questions  identified  in  the  self-assessment  are  shown  as  appendix 
2.  Specific  objectives  of  the  self-assessment  were  to; 

•  determine  the  extent  to  which  a  change  in  behavior  occurred  because  of  the  training  as  ec- 
plained  in  Kirkpatricks  Level  III  Evaluation-Behavior 

•  determine  students  perceived  ability  to  complete  tasks  after  training 

•  help  instmctors  determine  the  strengths  and  needs  of  students 

•  assist  students  in  improving  their  individual  academic  skills 

•  assist  in  predicting  if  a  change  in  behavior  will  occur  during  the  first  opportunity  to  utilize  the 
skill  set 

5.4.5. 1  Data  Gathering  Procedures  of  the  Self  Assessment 

•  Upon  completion  of  training,  paper  copies  of  the  rating  form  were  distributed  to  the  students. 

•  Rating  forms  identify  key  tasks.  Students  were  asked  to  annotate  how  much  assistance  they 
thought  they  needed  to  perform  each  task 
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6.0— PRESENTATION  OF  DATA 

The  EOC  survey  contains  30  questions  for  each  of  the  operator,  system  administrator  and  network 
administrator  courses.  Individual  responses  for  all  30  survey  questions  for  operator,  system  administrator, 
and  PSS  courses  are  not  listed  in  this  report  due  to  size  and  length  of  files.  Instead,  only  the  responses  to 
questions  that  assist  in  determining  the  effectiveness  of  the  training  and  those  that  directly  answer  the 
SPD’s  stated  research  questions  are  presented.  In  particular,  responses  to  the  following  are  reported: 

•  amount  of  TBMCS  experience 

•  amount  of  Legacy  experience 

•  facilitator’s  knowledge  of  the  overall  course 

•  course  provided  training  necessary  to  enable  students  to  complete  the  TBMCS  software  appli¬ 
cation  tasks  specific  to  their  work  center. 

•  overall  the  course  met  expectations 

•  length  of  course  was  appropriate 

•  classroom  environment 

6.1  TOTAL  FORCE  LEVEL  OPERATORS 

A  sample  of  n  =  293  for  Air  Force,  force-level  operators  participated  in  the  courses.  A  total  of  148 
Air  Force  surveys  were  received.  Response  rate  was  50%.  A  sample  of  n  =  146  Marine  force-level  qa- 
erators  participated  in  the  courses.  A  total  of  124  Marine  surveys  were  received.  Response  rate  was  84%. 
A  sample  of  n  =  6  for  Navy  force-level  operators  participated  in  the  courses.  A  total  of  6  Navy  surveys 
were  received.  Response  rate  was  100%.  A  sample  of  n  =  29  for  NORAD  force-level  operators  partici¬ 
pated  in  the  courses.  A  total  of  20  surveys  were  received.  Response  rate  was  69%.  Total  multi-service 
course  participation  was  474.  Total  multi-service  response  rate  was  63%.  This  is  a  sufficient  response  rate 
for  this  report. 

6.1.1  Force  Level  Operator  Responses  to  Surveys 

Tables  6.1-1  through  6.1-9  represent  operator  responses  to  survey  questions 
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Table  6.1-1.  TBMCS  Multi-Service  Responses  for  Legacy  Experience 


Force  Level  Operator  Training 

Legacy  Experience 

Students  Trained:  474  Surveys  Submited:  304 


■  Air  Force 

■  Marine  Corps 

□  Navy 

□  NORAD 


0  1-6  7-12  13-1-  Did  Not 

Answer 


Note:  Table  6.1-1  reveals  operator  legacy  experience.  An  average  of  23.02%  of  the  students  did  not  re¬ 
spond  to  this  question.  An  average  of  41  %  of  the  students  had  no  legacy  operator  experience.  An  average 
of  18.3%  of  the  students  possessed  1-6  months  legacy  experience.  An  average  of  5.05%  of  the  students 
possessed  7-12  months  legacy  experience.  An  average  of  12.54%  of  the  students  possessed  over  13 
months  legacy  experience.  With  a  cumulative  total  of  64.35  %  of  students  who  possessed  less  than  12 
months  legacy  experience  compared  to  12.54%  of  students  who  did  possess  legacy  experience  it  is  appar¬ 
ent  that  most  students  did  not  meet  the  minimum  course  prerequisite  of  one  year  legacy  experience  prior 
to  attending  the  course. 
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Table  6.1-2,  TBMCS  Multi-Service  Responses  for  TBMCS  Experience 


Force  Level  Operator  Training 

TBMCS  Experience 

Students  Trained:  474  Surveys  Submited:  304 


C7) 

ra 


u 

0) 

Q. 


■  Air  Force 

■  Marine  Corps 

□  Navy 

□  NORAD 


0  1-6  7-12  13-1-  Did  Not 

Answer 


Note:  Table  6.1-2  reveals  operator  TBMCS  experience.  An  average  of 23.02%  of  the  students  did  not  re¬ 
spond  to  this  question.  An  average  of  53.2%  of  the  students  had  no  TBMCS  experience.  An  average  of 
21.02%  of  the  students  possessed  1-6  months  TBMCS  experience.  An  average  of  7.01%  of  the  students 
possessed  7-12  months  legacy  experience.  An  average  of  .25%  of  the  students  possessed  over  12  months 
TBMCS  experience.  With  a  cumulative  total  of  81.23  %  of  students  who  possessed  less  than  12  months 
TBMCS  it  is  apparent  that  most  students  did  not  meet  the  minimum  course  prerequisite  of  one  year 
TBMCS  experience  prior  to  attending  the  course. 
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Table  6.1-3.  TBMCS  Multi-Service  Responses  for  Facilitator  Knowledge 


Force  Level  Operator  Training 

The  course  facilitator  was  knowledgeable  about  the  overall  course. 

Students  Trained:  474  Surveys  Submited:  298 


■  Air  Force 

■  Marine  Corps 
n  Navy 

□  NORAD 


Strongly  Disagree  Agree  Strongly  Did  Not 
Disagree  Agree  Answer 


Note:  Table  6. 1  -3  reveals  force  level  operator  perceptions  regarding  facilitator  knowledge  about  the 
overall  course.  An  average  of  31.8%  of  the  students  did  not  respond  to  this  question.  An  average  of 
37.1%  of  the  students  strongly  agreed  that  the  facilitator  was  knowledgeable  of  the  overall  course.  An 
average  of  36.8%  of  the  students  agreed  that  the  facilitator  was  knowledgeable  of  the  overall  course.  An 
average  of  1.2%  strongly  disagreed  that  the  instructor  was  knowledgeable  of  the  overall  course.  An  aver¬ 
age  of  2.6%  disagreed  that  the  instructor  was  knowledgeable  of  the  overall  course.  With  a  cumulative 
total  of  73. 9%  of  students  who  strongly  agree  or  agree  that  the  instructor  was  knowledgeable  about  the 
subject  matter  versus  a  cumulative  total  of  3.2%  of  students  who  strongly  disagree  or  disagree  that  the 
instructor  is  knowledgeable  about  the  overall  course,  this  information  suggests  that  the  students  per¬ 
ceived  the  course  instructors  to  be  highly  knowledgeable  of  the  course  content. 
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Table  6.1-4,  TBMCS  Multi-Service  Responses  for  TBMCS  Work  Center  Training 


Force  Level  Operator  Training 

I  believe  that  the  course  provided  the  training  necessary  to  enable 
me  to  complete  the  TBMCS  software  application  tasks  specific  to 
my  work  center. 

Students  Trained:  474  Surveys  Submited:  302 


Strongly  Disagree  Agree  Strongly  Did  Not 
Disagree  Agree  Answer 


Note:  Table  6.1-4  reveals  force  level  operator  perceptions  regarding  TBMCS  training  applicability  spe¬ 
cific  to  an  operator’s  duty  center.  An  average  of  23.2%  of  the  students  did  not  respond  to  this  question. 
An  average  of  10.6%  of  the  students  strongly  agreed  that  the  training  provided  was  specific  to  their  duty 
center.  An  average  of 44.02%  of  the  students  agreed  that  the  training  provided  was  specific  to  their  duty 
center.  An  average  of  1.02%  strongly  disagreed  that  the  training  provided  was  specific  to  their  duty  cen¬ 
ter.  An  average  of  21.15%  disagreed  that  the  training  provided  was  specific  to  their  duty  center.  With  a 
cumulative  total  of  54.62%  of  students  who  strongly  agree  or  agree  that  the  training  provided  was  spe¬ 
cific  to  their  duty  center  versus  a  cumulative  total  of  22.17%  of  students  who  strongly  disagree  or  dis¬ 
agree  that  that  the  training  provided  was  specific  to  their  duty  center,  this  information  suggests  that  most 
students  perceived  that  the  training  provided  was  specific  to  their  duty  center. 
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Table  6.1-5.  TBMCS  Multi-Service  Responses  for  Course  Expectation 


Force  Level  Operator  Training 

Overall,  the  course  met  my  expectations. 

Students  Trained:  474  Surveys  Submited:  301 


■  Air  Force 

■  Marine  Corps 

□  Navy 

□  NORAD 


Strongly  Disagree  Agree  Strongly  Did  Not 
Disagree  Agree  Answer 


Note:  Table  6. 1  -5  reveals  force  level  operator  expectations  about  the  overall  course.  An  average  of 
24.07%  of  the  students  did  not  respond  to  this  question.  An  average  of  10.92%  of  the  students  strongly 
agreed  that  their  course  expectations  were  met.  An  average  of  42.65%  of  the  students  agreed  that  their 
course  expectations  were  met.  An  average  of  .95%  strongly  disagreed  that  their  course  expectations  were 
met.  An  average  of  21.07%  disagreed  that  their  course  expectations  were  met.  With  a  cumulative  total  of 
53.57%  of  students  who  strongly  agree  or  agree  that  their  course  expectations  were  met  versus  a  cumu¬ 
lative  total  of  22.02%  of  students  who  strongly  disagree  or  disagree  that  their  course  expectations  were 
met,  this  information  suggests  that  most  students  perceived  that  their  course  expectations  were  met. 
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Table  6.1-6.  TBMCS  Multi-Service  Responses  for  Course  Length 


Force  Level  Operator  Training 

The  length  of  this  course  was  appropriate. 

Students  Trained:  474  Surveys  Submited:  301 


■Air  Force 

■  Marine  Corps 

■  Navy 
□NORAD 


Note:  Table  6.1-6  reveals  force  level  operator  perceptions  about  the  overall  course  length.  An  average  of 
23.37%  of  the  students  did  not  respond  to  this  question.  An  average  of  2.72%  of  the  students  strongly 
agreed  that  the  length  of  the  course  was  appropriate.  An  average  of  35.67%  of  the  students  agreed  that 
the  length  of  the  course  was  appropriate.  An  average  of  2.55%  strongly  disagreed  that  the  length  of  the 
course  was  appropriate.  An  average  of  35.67%  disagreed  that  the  length  of  the  course  was  appropriate. 
The  cumulative  total  of  38.39%  of  students  who  strongly  agree  or  agree  that  the  length  of  the  course  was 
appropriate  versus  a  cumulative  total  of  38.22%  of  students  who  strongly  disagree  or  disagree  that  the 
length  of  the  course  was  appropriate  suggests  that  there  was  equal  “disagreement"  on  the  course  length, 
thus  the  length  of  the  course  should  be  reviewed  prior  to  the  next  fielding. 
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Table  6.1-7.  TBMCS  Multi-Service  Responses  for  Class  Environment 


Force  Level  Operator  Training 

Classroom  Environment 


(U 

O) 

(0 

c 

(U 

o 

<u 


□  Was  Acceptable 

■  Needs  Improvement 

□  Did  Not  Respond 


The 

reliability  of 
the 

equipment 
used  for  the 
course 


The 

classroom 

temperature 


The  pysical 
work  space 


Note:  Table  6.1-7  reveals  force  level  operator  perceptions  about  the  course  environment  (equipment, 
network  connection,  temperature,  noise  level,  and  work  space).  An  average  of 29.84%  of  the  students  did 
not  respond  to  this  question.  An  average  of  56.74%  of  the  students  agreed  that  the  course  environment 
was  acceptable.  An  average  of  13.54%  of  the  students  agreed  that  the  course  environment  needed  im¬ 
provement. 
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Table  6.1-8.  TBMCS  Multi-Service  Responses  for  Pre/Post  Test 


Note:  Table  6.1-8  reveals  differences  between  service  pre  and  post  test  scores.  AF  averaged  an  8%  h- 
crease,  MC  averaged  a  41%  increase,  Navy  averaged  a  30%  increase,  and  NORAD  averaged  a  52%  in¬ 
crease.  Cumulative  pre  test  average  was  54.87%.  Cumulative  post  test  score  was  87.62%.  Cumulative 
average  gain  was  32. 7%.  Student  test  questions  are  identified  as  learning  objectives  as  the  course  is  de¬ 
signed.  With  all  students  shifting  from  a  below  average  score  (<75%)  to  above  average  it  can  be  pre¬ 
sumed  that  learning  objectives  were  met  as  a  result  of  the  instruction. 
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Table  6.1-9,  TBMCS  Multi-Service  Responses  for  Force  Level  Operator  Self  Assessment 


Overall  Performance 
22  Jan  to  29  Jun  2001 


60 

50 

40 

30 

20 

10 

0 


1.  Could  not  2.  Could  3.  Could  4.  Could  5.  Not 
accomplish  accomplish  accomplish  accomplish  attempted 

w/over-the-  w/on-  without  help 

shoulder  llnehelp 

help 


Note:  Table  6.1-9  reveals  operator  self-assessment  data.  A  sample  of  n  =  248  total  students  participated 
in  the  operator  course.  Total  ops/plans  key  tasks  evaluated  =  66.  Total  possible  responses  =  16,949.  To¬ 
tal  responses  received  =  7,645.  Response  rate  =  45%.  Of  the  45%  who  responded,  1.4%  stated  they  could 
not  accomplish  the  tasks,  15.3%  stated  they  could  accomplish  with  over-the-shoulder  help,  24.1%  stated 
they  could  accomplish  with  on-line  help,  52%  stated  they  could  accomplish  without  help,  and  7.3%  did 
not  attempt  the  task.  With  76.1%  stating  that  they  could  accomplish  the  tasks  by  themselves  versus  1.4% 
who  stated  they  could  not  accomplish  the  task,  this  information  suggests  that  a  significant  portion  of  the 
operators  can  use  TBMCS  independently. 


6.2  SYSTEMS  ADMINISTRATOR 

A  sample  of  n  =  62  for  AF  force  level  system  administrators  participated  in  the  courses.  A 
total  of  35  AF  surveys  were  received.  Response  rate  was  56%.  A  sample  of  n  =  98  Marine  force 
level  system  administrators  participated  in  the  courses.  A  total  of  82  Marine  surveys  were  re¬ 
ceived.  Response  rate  was  84%.  A  total  of  n=9  NORAD  force  level  system  administrators  par¬ 
ticipated  in  the  courses.  A  total  of  9  surveys  were  received.  Response  rate  was  100%.  Navy  force 
level  system  administrators  did  not  participate  in  the  courses. 

6.2.1  System  Administrator  Responses  to  Surveys 

Tables  6.2-1  through  6.2-8  represent  system  administrator  responses  to  survey  questions. 
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Table  6.2-1,  TBMCS  Multi-Service  Responses  for  Legacy  Experience 

Force  Level  SysAd  Training 

Legacy  Experience 

Students  Trained:  169  Surveys  Submited:  126 


50 


0  1-6  7-12  13-1-  Did  Not 

Answer 


Note:  Table  6.2-1  reveals  system  administrator  legacy  experience.  An  average  of  19.6%  of  the  students 
did  not  respond  to  this  question.  An  average  of 28.96%  of  the  students  had  no  legacy  system  administra¬ 
tor  experience.  An  average  of  15.4%  of  the  students  possessed  1-6  months  legacy  experience.  An  average 
of  4.26%  of  the  students  possessed  7-12  months  legacy  experience.  An  average  of  31.43%  of  the  students 
possessed  over  12  months  legacy  experience.  With  a  cumulative  total  of  48.96  %  of  students  who  pos¬ 
sessed  less  than  12  months  of  legacy  experience  compared  to  28.96%  of  students  who  possessed  more 
thanl2  months  of  legacy  experience  it  is  apparent  that  most  students  did  not  meet  the  minimum  course 
prerequisite  of  one  year  legacy  experience  prior  to  attending  the  course. 
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Table  6.2-2  TBMCS  Multi-Service  Responses  for  TBMCS  Experience 


Force  Level  SysAd  Training 

TBMCS  Experience 

Students  Trained:  169  Surveys  Submited:  126 


Note:  Table  6.2-2  reveals  system  administrator  TBMCS  experience.  An  average  of  19.93%  of  the  students 
did  not  respond  to  this  question.  An  average  of  29.5%  of  the  students  had  no  TBMCS  experience.  An  av¬ 
erage  of  36.2%  of  the  students  possessed  1-6  months  TBMCS  experience.  An  average  of  5.23%  of  the  stu¬ 
dents  possessed  7-12  months  legacy  experience.  An  average  of  8.8%  of  the  students  possessed  over  12 
months  TBMCS  experience.  With  a  cumulative  total  of  70.93  %  of  students  who  possessed  0-12  months 
TBMCS  experience  compared  to  8.8%  of  students  who  possessed  more  than  12  months  TBMCS  experi¬ 
ence  it  is  apparent  that  most  students  did  not  meet  the  minimum  course  prerequisite  of  one  year  TBMCS 
experience  prior  to  attending  the  course. 
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Table  6.2-3.  TBMCS  Multi-Service  Responses  for  Facilitator  Knowledge 


Force  Level  SysAd  Training 

The  course  facilitator  was  knowledgeable  about  the  overall  course. 

Students  Trained:  169  Surveys  Submited:  126 
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Strongly  Disagree  Agree  Strongly  Did  Not 
Disagree  Agree  Answer 


Note:  Table  6.2-3  reveals  system  administrator  perceptions  regarding  facilitator  knowledge  about  the 
overall  course.  An  average  of  20.16%  of  the  students  did  not  respond  to  this  question.  An  average  of 
27.06%  of  the  students  strongly  agreed  that  the  facilitator  was  knowledgeable  of  the  overall  course.  An 
average  of 50.06%  of  the  students  agreed  that  the  facilitator  was  knowledgeable  of  the  overall  course.  An 
average  of  .6%  strongly  disagreed  that  the  instructor  was  knowledgeable.  An  average  of  2.03%  disagreed 
that  the  instructor  was  knowledgeable.  With  a  cumulative  total  of  77.12%  of  students  who  strongly  agree 
or  agree  that  the  instructor  was  knowledgeable  about  the  subject  matter  versus  a  cumulative  total  of 
2.63%  of  students  who  strongly  disagree  or  disagree  that  the  instructor  was  knowledgeable  about  the 
overall  course,  this  information  suggests  that  the  students  perceived  the  course  instructors  to  be  highly 
knowledgeable  of  the  course  content. 
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Table  6.2-4.  TBMCS  Multi-Service  Responses  for  Work  Center  Training 


Force  Level  SysAd  Training 

I  believe  that  the  course  provided  the  training  necessary  to  enable 
me  to  complete  the  TBMCS  software  application  tasks  specific  to 
my  work  center. 

Students  Trained:  169  Surveys  Submited:  126 


■  Air  Force 

■  Marine  Corps 
□  NORAD 


Strongly  Disagree  Agree  Strongly  Did  Not 
Disagree  Agree  Answer 


Note:  Table  6.2.4  reveals  force  level  system  administrator  perceptions  regarding  TBMCS  training  appli¬ 
cability  specific  to  an  system  administrator’s  duty  center.  An  average  of  20.16%  of  the  students  did  not 
respond  to  this  question.  An  average  of  16.86%  of  the  students  strongly  agreed  that  the  training  provided 
was  specific  to  their  duty  center.  An  average  of  48%  of  the  students  agreed  that  the  training  provided  was 
specific  to  their  duty  center.  An  average  of  1.3%  strongly  disagreed  that  the  training  provided  was  spe¬ 
cific  to  their  duty  center.  An  average  of  13.56%  disagreed  that  the  training  provided  was  specific  to  their 
duty  center.  With  a  cumulative  total  of  64.86%  of  students  who  strongly  agree  or  agree  that  the  training 
provided  was  specific  to  their  duty  center  versus  a  cumulative  total  of  14.86%  of  students  who  strongly 
disagree  or  disagree  that  that  the  training  provided  was  specific  to  their  duty  center,  this  information 
suggests  that  the  majority  of  students  perceived  that  the  training  provided  was  specific  to  their  duty  cen¬ 
ter. 
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Table  6.2-5.  TBMCS  Multi-Service  Responses  for  Course  Expectation 


Force  Level  SysAd  Training 

Overall,  the  course  met  my  expectations. 

Students  Trained:  169  Surveys  Submited:  126 


■  Air  Force 

■  Marine  Corps 
□  NORAD 


Strongly  Disagree  Agree  Strongly  Did  Not 
Disagree  Agree  Answer 


Note:  Table  6.2-5  reveals  force  level  system  administrator  expectations  about  the  overall  course.  An  av¬ 
erage  of  19.9%  of  the  students  did  not  respond  to  this  question.  An  average  of  12.7%  of  the  students 
strongly  agreed  that  their  course  expectations  were  met.  An  average  of  57.06%  of  the  students  agreed 
that  their  course  expectations  were  met.  An  average  of  1.5%  strongly  disagreed  that  their  course  expec¬ 
tations  were  met.  An  average  of  8.73%  disagreed  that  their  course  expectations  were  met.  With  a  cumu¬ 
lative  total  of  69.76%  of  students  who  strongly  agree  or  agree  that  their  course  expectations  were  met 
versus  a  cumulative  total  of  10.23%  of  students  who  strongly  disagree  or  disagree  that  their  course  ec- 
pectations  were  met,  this  information  suggests  that  the  majority  of  students  perceived  that  their  course 
expectations  were  met. 
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Table  6.2-6.  TBMCS  Multi-Service  Responses  for  Length  of  Course 


Force  Level  SysAd  Training 

The  length  of  this  course  was  appropriate. 

Students  Trained:  169  Surveys  Submited:  126 
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Note:  Table  6.2-6  reveals  force  level  system  administrators  perceived  about  the  overall  course  length.  An 
average  of  20.3%  of  the  students  did  not  respond  to  this  question.  An  average  of  8.93%  of  the  students 
strongly  agreed  that  the  length  of  the  course  was  appropriate.  An  average  of  47.3%  of  the  students 
agreed  that  the  length  of  the  course  was  appropriate.  An  average  of  5.76%  strongly  disagreed  that  the 
length  of  the  course  was  appropriate.  An  average  of  17.7%  disagreed  that  the  length  of  the  course  was 
appropriate.  With  a  cumulative  total  of 56.23%  of  students  who  strongly  agree  or  agree  that  the  length  of 
the  course  was  appropriate  versus  a  cumulative  total  of 23.46%  of  students  who  strongly  disagree  or  dis¬ 
agree  that  the  length  of  the  course  was  appropriate,  this  information  suggests  that  most  students  per¬ 
ceived  that  the  length  of  the  course  was  appropriate. 
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Table  6.2-7.  TBMCS  Multi-Service  Responses  for  Classroom  Environment 


Force  Level  SysAd  Training 

Classroom  Environment 
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□  Did  Not  Respond 


The  The  The  The 

reliability  of  reliability  of  classroom  classroom 

the  network  temperature  niose  level 

equipment  connection 
used  for  the  for  the 

course  course 


The  pysical 
work  space 


Note:  Table  6.2-7  reveals  force  level  system  administrator  perceptions  about  the  course  environment 
(equipment,  network  connection,  temperature,  noise  level,  and  work  space).  An  average  of 20.74%  of  the 
students  did  not  respond  to  this  question.  An  average  of  68.84%  of  the  students  agreed  that  the  course 
environment  was  acceptable.  An  average  of  10.42%  of  the  students  agreed  that  the  course  environment 
needed  improvement. 
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Table  6.2-8.  TBMCS  Multi-Service  Responses  for  Pre/Post  Test 


Note:  Table  6.2-8  reveals  differences  between  pre  and  post  test  scores  for  system  administrators.  AF  av¬ 
eraged  a  31%  increase  and  Marine  Corps  averaged  a  53%  increase.  Navy  and  NORAD  did  not  partici¬ 
pate.  Cumulative  pre  test  average  was  45.5%.  Cumulative  post  test  score  was  87.5%.  Cumulative  average 
gain  was  42%.  Student  test  questions  are  identified  as  learning  objectives  as  the  course  is  designed.  With 
all  students  shifting  from  below  average  score  (>75%)  to  above  average  (<75%)  it  can  be  presumed  that 
learning  objectives  were  met  as  a  result  of  the  instruction. 

6.3  PERIMETER  SECURITY  SYSTEM 

A  sample  of  n  =  88  for  AF  force  level  PSS  network  administrators  participated  in  the  courses.  A  total 
of  69  AF  surveys  were  received.  Response  rate  was  78%.  A  sample  of  n  =  61  Marine  Corps  PSS  network 
administrators  participated  in  the  courses.  A  total  of  38  Marine  Corps  surveys  were  received.  Response 
rate  was  62%.  A  sample  of  n  =  28  for  Navy  PSS  network  administrators  participated  in  the  courses.  A 
total  of  18  Navy  surveys  were  received.  Response  rate  was  69%.  Total  multi-service  course  participation 
was  171.  Total  multi-service  response  rate  was  73%. 

6.3.1  Perimeter  Security  System  responses  to  Questions 

Tables  6.3-1  through  6.3-5  represent  network  administrators’  responses  to  survey  questions 
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Table  6.3-1.  TBMCS  Multi-Service  Responses  for  Facilitator  Knowledge 


Force  Level  PSS  Training 

The  course  facilitator  was  knowledgeable  about  the  overall 

course. 


Students  Trained:  175  Surveys  Submited:  125 


Strongly  Disagree  Agree  Strongly  Did  Not 
Disagree  Agree  Answer 


Note:  Table  6.3-1  reveals  perimeter  system  security  network  administrator  perceptions  regarding  facili¬ 
tator  knowledge  about  the  overall  course.  An  average  of  30.20%  of  the  students  did  not  respond  to  this 
question.  An  average  of  41.73%  of  the  students  strongly  agreed  that  the  facilitator  was  knowledgeable  of 
the  overall  course.  An  average  of  24.9%  of  the  students  agreed  that  the  facilitator  was  knowledgeable  of 
the  overall  course.  An  average  of  .8%  strongly  disagreed  that  the  instructor  was  knowledgeable.  An  av¬ 
erage  of  2.2%  disagreed  that  the  instructor  was  knowledgeable.  With  a  cumulative  total  of 66.63%  of  stu¬ 
dents  who  strongly  agree  or  agree  that  the  instructor  was  knowledgeable  about  the  subject  matter  versus 
a  cumulative  total  of  3.06%  of  students  who  strongly  disagree  or  disagree  that  the  instructor  is  knowl¬ 
edgeable  about  the  overall  course,  this  information  suggests  that  the  students  perceived  the  course  in¬ 
structors  to  be  highly  knowledgeable  of  the  course  content. 
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Table  6.3-2,  TBMCS  Multi-Service  Responses  for  TBMCS  Work  Center  Training 


Force  Level  PSS  Training 

I  believe  that  the  course  provided  the  training  necessary  to  enable 
me  to  complete  the  TBMCS  software  application  tasks  specific  to 
my  work  center. 

Students  Trained:  175  Surveys  Submited:  123 


■  Air  Force 

■  Marine  Corps 
□  Navy 


Strongly  Disagree  Agree  Strongly  Did  Not 
Disagree  Agree  Answer 


Note:  Table  6.3-2  reveals  force  perimeter  security  system  network  administrator’s  perceptions  regarding 
TBMCS  training  applicability  specific  to  an  their  duty  center.  An  average  of  31.3%  of  the  students  did  not 
respond  to  this  question.  An  average  of  20.8%  of  the  students  strongly  agreed  that  the  training  provided 
was  specific  to  their  duty  center.  An  average  of  42.1%  of  the  students  agreed  that  the  training  provided 
was  specific  to  their  duty  center.  An  average  of  .33%  strongly  disagreed  that  the  training  provided  was 
specific  to  their  duty  center.  An  average  of  5. 7%  disagreed  that  the  training  provided  was  specific  to  their 
duty  center.  With  a  cumulative  total  of  62. 9%  of  students  who  strongly  agree  or  agree  that  the  training 
provided  was  specific  to  their  duty  center  versus  a  cumulative  total  of  6.03%  of  students  who  strongly 
disagree  or  disagree  that  that  the  training  provided  was  specific  to  their  duty  center,  this  information 
suggests  that  the  majority  of  students  perceived  that  the  training  provided  was  specific  to  their  duty  cen¬ 
ter. 
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Table  6.3-3.  TBMCS  Multi-Service  Responses  for  TBMCS  Work  Center  Training 


Force  Level  PSS  Training 

Overall,  the  course  met  my  expectations. 

Students  Trained:  175  Surveys  Submited:  124 


Strongly  Disagree  Agree  Strongly  Did  Not 
Disagree  Agree  Answer 


Note:  Table  6.3-3  reveals  force  level  perimeter  system  security  network  administrator  expectations  about 
the  overall  course.  An  average  of  30%  of  the  students  did  not  respond  to  this  question.  An  average  of 
26.8%  of  the  students  strongly  agreed  that  their  course  expectations  were  met.  An  average  of  38.9%  of 
the  students  agreed  that  their  course  expectations  were  met.  An  average  of  0%  strongly  disagreed  that 
their  course  expectations  were  met.  An  average  of  4.26%  disagreed  that  their  course  expectations  were 
met.  With  a  cumulative  total  of  65.7%  of  students  who  strongly  agree  or  agree  that  their  course  expecta¬ 
tions  were  met  versus  a  cumulative  total  of  4.26%  of  students  who  strongly  disagree  or  disagree  that  their 
course  expectations  were  met,  this  information  suggests  that  the  majority  of  students  perceived  that  their 
course  expectations  were  met 
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Table  6.3-4.  TBMCS  Multi-Service  Responses  for  Length  of  Course 


Force  Level  PSS  Training 

The  length  of  this  course  was  appropriate. 

Students  Trained:  175  Surveys  Submited:  123 


Strongly  Disagree  Agree  Strongly  Did  Not 
Disagree  Agree  Answer 


Note:  Table  6.3-4  reveals  force  perimeter  security  systems  network  administrators  perception  about  the 
overall  course  length.  An  average  of  31.06%  of  the  students  did  not  respond  to  this  question.  An  average 
of  19.53%  of  the  students  strongly  agreed  that  the  length  of  the  course  was  appropriate.  An  average  of 
39.9%  of  the  students  agreed  that  the  length  of  the  course  was  appropriate.  An  average  of  .86%  strongly 
disagreed  that  the  length  of  the  course  was  appropriate.  An  average  of  8.93%  disagreed  that  the  length  of 
the  course  was  appropriate.  With  a  cumulative  total  of  59.43%  of  students  who  strongly  agree  or  agree 
that  the  length  of  the  course  was  appropriate  versus  a  cumulative  total  of  9.79%  of  students  who  strongly 
disagree  or  disagree  that  the  length  of  the  course  was  appropriate,  this  information  suggests  that  most 
students  perceived  that  the  length  of  the  course  was  appropriate. 
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Table  6.3-5.  TBMCS  Multi-Service  Responses  for  Classroom  Environment 


Force  Level  PSS  Training 

Classroom  Environment 


the 

equipment 
used  for  the 
course 


network 
connection 
for  the 
course 


temperature  niose  level 


o  Was  Acceptable 
■  Needs  Improvement 
□  Did  Not  Respond 


Note:  Table  6.3-5  reveals  force  level  perimeter  security  system  network  administrator  perceptions  about 
the  course  environment  (equipment,  network  connection,  temperature,  noise  level,  and  work  space).  An 
average  of  41.18%  of  the  students  did  not  respond  to  this  question.  An  average  of 48.84%  of  the  students 
agreed  that  the  course  environment  was  acceptable.  An  average  of  9.9%  of  the  students  agreed  that  the 
course  environment  needed  improvement. 
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7.0— FINDINGS 

A  major  goal  of  this  study  is  to  determine  the  impact  of  training.  For  purposes  of  this  study  impact  is 
viewed  as  “measurable  learning”  and  “student  perception”  of  learning.  Perception  drives  motivation  and 
emotion.  Emotion  drives  attention  and,  in  turn,  memory.  Abell  (2000)  stated  emotion  is  often  a  more 
powerful  influence  on  behavior  than  logic.  Thus,  it  is  an  important  indicator  of  course  satisfaction.  The 
research  questions  below  have  been  supported  with  quantitative  data  from  the  end  of  course  critique,  pre- 
/post-test  scores,  focus  groups  and  observable  behavior  by  the  program  office. 

Research  Question  1:  Were  the  majority  of  students  satisfied  at  the  completion  of  training? 

Data  revealed  in  tables  6.1-5,  6.2-5  and  6.3-3  pertaining  to  operator,  system  administrator  and  PSS 
course  expectations  indicate  that  53.57%  operators,  69.76  of  system  administrators,  and  65.7%  of  PSS 
network  administrators  agreed  that  their  expectations  were  met.  Although  the  data  reflect  the  majority  of 
students  being  satisfied,  the  range  is  low,  which  means  that  training  can  continue  to  improve.  Qualitative 
information  from  focus  groups  and  observations  conducted  by  the  SPO  reflected  that  many  students  had 
pre-conceived  and/or  negative  attitudes  in  regard  to  the  training.  A  common  misconception  was  that  the 
“TBMCS  system”  was  unstable  and  difficult  to  use.  Common  observable  negative  attitudes  appeared 
when  students  were  required  to  learn  via  web-based  training  instead  of  instructor  led.  Students  were  often 
hostile,  had  short  attention  spans,  and  showed  resistance  to  learn  without  the  instructor.  Additionally, 
when  the  instructor-led  approach  was  used  many  students  had  “anti-contractor”  preconceptions,  which 
interfered  with  learning.  Observable  behaviors  such  as  increased  persistence  and  voluntary  engagement  in 
the  task  was  seldom  noticed  by  the  Program  Office. 

Research  Question  2:  Will  there  be  a  difference  in  the  students  test  scores  after  completing  the  train¬ 
ing? 

Cumulative  average  gain  for  operators  and  system  administrators  was  37.5%.  Student  test  questions 
are  identified  as  learning  objectives  as  the  course  is  designed.  With  all  students  shifting  from  below  aver¬ 
age  score  {<15%)  to  above  average  (>75%)  it  can  be  presumed  that  learning  objectives  were  met  as  a  re¬ 
sult  of  the  instmction. 

Research  Question  3:  Will  users  be  confident  in  their  ability  to  perform  key  tasks  upon  completion  of 
the  training? 

The  student  self-assessment  instmment  was  used  to  determine  the  confidence  level  of  the  users  in 
performing  key  tasks.  Response  rate  of  the  student  self-assessment  was  45%.  Of  the  45%  who  responded, 
1.4%  stated  they  could  not  accomplish  the  key  tasks,  91.4%  stated  they  could  complete  the  key  tasks  with 
over- the- shoulder  help,  on-line  help,  or  without  help,  leaving  7.3%  who  did  not  attempt  the  task  after 


49 


Training  Program  Review 
TBMCS  Training 
Program  Evaluation  v.  1.0.1 

training.  Students’  perception  of  their  ability  to  perform  key  tasks  is  high.  The  key  to  maintaining  this 
perception  is  to  refresh  these  skills  with  OJT  and  continuous  exercises. 

Research  Question  4:  Is  there  a  correlation  between  user  experience  and  EOC  satisfaction? 

Data  in  Table  6.1-1  indicate  64.35  %  of  force-level  operators  possessed  less  than  the  required  12 
months  legacy  experience  compared  to  12.54%  of  operators  who  did  possess  the  course  prerequisite  of  12 
months  legacy  experience.  Data  in  Table  6.1-5  indicate  that  53.57%  operators  stated  the  course  met  their 
expectations.  Data  revealed  in  Table  6.2-lindicate  an  average  of  48.96%  of  system  administrator  students 
possessed  less  than  12  months  of  required  legacy  experience  compared  to  31.43%  of  system  administra¬ 
tors  who  did  possess  the  12-month  legacy  experience  prerequisite.  Data  in  Table  6.2-5  indicate  that 
69.76%  of  system  administrators  stated  the  course  met  their  expectations.  Although  the  majority  of  q)- 
erators  and  system  administrators  did  not  meet  the  required  course  prerequisites  of  12  months  of  legacy 
experience,  there  appears  to  be  a  correlation  between  higher  experience  and  higher  course  satisfaction  as 
indicated  by  the  system  administrators. 

Research  Question  5:  Will  the  students  perceive  the  facilitator  as  knowledgeable  about  the  course 
content? 

A  cumulative  total  of  72.55%  of  operator,  system  administrator  and  PSS  students  strongly  agreed  or 
agreed  that  the  instructor  was  knowledgeable  about  the  subject  matter  versus  a  cumulative  total  of  3.15% 
of  students  who  strongly  disagreed  or  disagreed  that  the  instmctor  was  knowledgeable  about  the  overall 
course.  This  information  reveals  that  the  majority  of  students  perceived  the  course  instmctors  to  be  highly 
knowledgeable  of  the  course  content.  Observations  and  focus  groups  indicated  that  students  were  frus¬ 
trated  with  facilitator  lack  of  knowledge  when  method  of  instmction  was  100%  web  based;  however,  in¬ 
stmctor  credibility  increased  when  method  of  instmction  changed  to  instmctor-led  practical  exercises. 

Research  Question  6:  Will  students  perceive  that  the  course  covered  the  key  TBMCS  skills  specific  to 
their  work  center? 

Data  in  Tables.6.1-4,  6.2-4,  and  6.4-3  reveal  that  a  cumulative  total  of  61.73%  of  students  strongly 
agreed  or  agreed  that  the  training  provided  was  specific  to  their  duty  center  versus  a  cumulative  total  of 
13.92%  of  students  who  strongly  disagreed  or  disagreed  that  the  training  provided  was  specific  to  their 
duty  center.  This  information  reveals  that  most  students  perceived  the  training  provided  was  specific  to 
their  duty  center.  Observations  and  focus  groups  indicate  a  correlation  between  user  experience  and  user 
perception  of  the  training  being  specific  to  their  work  center.  Those  students  who  did  not  meet  the  12- 
month  legacy  experience  prerequisite  exhibited  a  lack  of  understanding  of  the  TBMCS  system,  data  flows 
between  cells,  and  how  their  duty  position  related  to  the  tasks  and  skills  supported  by  TBMCS. 
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Research  Question  7:  Will  students  perceive  that  their  units  provided  a  workspace  that  supported  a 
successful  training  environment? 

Cumulative  perceptions  about  the  course  environment  (equipment,  network  connection,  temperature, 
noise  level,  workspace)  revealed  an  average  of  57.93%  of  the  students  agreed  that  the  course  environment 
was  acceptable.  An  average  of  11.31%  of  the  students  agreed  that  the  course  environment  needed  in- 
provement.  Focus  groups  and  observations  revealed  that  students  were  often  unhappy  about  the  physical 
work  environment  due  to  lack  of  air  conditioning,  close  proximity  of  other  students,  and  high  noise. 
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8.0— BARRIERS  AND  ISSUES 

A  major  goal  of  this  study  is  to  provide  an  in-depth  analysis  to  assist  future  C2  decision  makers  in 
determining  what  conditions  distributed  learning  is  likely  to  be  effective  for  future  C2  systems.  Based  on 
the  experience  in  managing  TBMCS  training,  ESC  has  identified  the  following  barriers  and  issues  as  ob¬ 
stacles  for  an  effective  implementation  of  a  distributed  learning  environment.  It  is  recommended  that  C2 
program  managers  understand  the  respective  impacts  and  consequences  of  these  limitations  as  part  of 
their  decision  making  progress  when  allocating  training  budgets,  identifying  resources,  and  establishing 
processes. 

•  Funding.  A  significant  issue  in  providing  effective  training  is  the  inconsistency  of  funds  allo¬ 
cated  for  training.  On  average  10%  of  the  total  TBMCS  software  development  program  was  al¬ 
located  to  training  over  the  life  of  the  contract.  During  Program  Objective  Memorandum  (POM) 
and  Amended  Program  Objective  Memorandum  (APOM)  reviews  funding  was  often  removed 
from  the  training  budget  due  to  management  misconception  and  lack  of  understanding  of  the  as¬ 
sociated  cost  and  the  importance  of  designing,  developing,  and  implementing  web-based  train¬ 
ing. 

•  Evaluation.  Multi-service  criteria  and  standards  for  evaluating  C2  weapon  system  specific 
training  were  not  established  for:  a)  summative  reviews  of  training  materials  and  course  impb- 
mentation,  b)  multi-service  test  activities  and  c)  OJT  training  plans. 

(1)  AF  Policy  Directive  (AFPD)  36-22  and  supporting  AFI  identify  the  design  and  develop¬ 
ment  of  training.  These  regulations  provide  a  structured  systems  design  framework  to 
identify  training  needs  and  to  design  the  most  effective  and  efficient  means  of  providing 
training.  These  regulations  are  limited  in  their  applicability  since  they  primarily  focus  on 
the  material  design  and  development  process.  A  systems/holistic  training  evaluation  sp- 
proach  to  include  a  summative  course  evaluation  is  needed  for  C2  managers  to  make 
judgments  regarding  the  current  training  program  and  decisions  for  future  training  pro¬ 
grams. 

(2)  Developmental  Test/Operational  Test  (DT/OT)  leads  did  not  identify  training  standards  or 
performance  criteria;  thus,  training  evaluations  during  test  events  are  unstructured  and  per¬ 
sonality  dependent.  A  deficiency  exists  in  obtaining  quantitative  data  to  validate  the  effec¬ 
tiveness  of  training. 

(3)  MTT  facilitators  have  identified  key  tasks  and  EOC  objectives  for  students.  For  most  stu¬ 
dents  TBMCS  is  a  wartime  application  and  is  not  used  during  peacetime  operations;  thus, 
when  returning  to  their  duty  station,  their  newly  learned  skill  sets  are  not  immediately 
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applied.  As  a  result,  skill  decay  takes  place.  Knowledge  retention  refers  to  the  e- 
membrance  of  facts,  terminology,  concepts,  and  rules  similar  to  the  knowledge 
level  of  Blooms’  (1956)  taxonomy.  The  variability  of  key  tasks  can  be  substantial  if  stu¬ 
dents  do  not  repeat  them  after  they  are  learned.  For  example,  in  field  research  with  soldiers 
who  have  been  recalled  to  service  after  being  away  for  more  than  a  year,  Sabol  and  Wisher 
(2001)  report  skill  losses  ranging  between  27%  and  83%  for  hands-on  tasks.  Such  proce¬ 
dural  tasks  at  the  application  level  are  of  major  concern  to  the  proficiency  of  TBMCS. 
Performance  criteria  for  TBMCS  users  are  not  documented  upon  completion  of  training.  Without 
adequate  performance  criteria  as  identified  in  Figure  8.0- 1 ,  the  training  program  at  whole  is  at  risk. 


Total  Training  to  be  Evainated  by  SDP,  OTAs,  and  Service  Commander 


Figure  8.0-1.  Performance  Criteria 

Lack  of  OJT  and  Continuation  Training  Plans .  Many  service  locations  lack  a  documented  training 
plan  identifying  continuation  training  requirements  and  training  manager  responsibilities.  When  a  training 
program  is  not  established,  or  when  a  training  manager  has  not  been  appointed  at  the  users  home  station, 
the  user  cannot  maintain  proficiency.  AFI  33-2201  identifies  training  Major  Command  (MAJCOM)  e- 
quirements.  Although  training  requirements  are  clearly  stated  in  Air  Force  regulations,  most  commanders 
have  not  implemented  an  OJT  program  that  includes  TBMCS  continuation  training  as  part  of  the 
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curriculum.  When  this  happens,  TBMCS  training  as  a  whole  is  affected  and  ESC  CAFC2  is  frequently 
identified  as  providing  poor  training. 

Technology  Planning.  A  key  lesson  learned  in  achieving  an  efficient  distributed  training  envircn- 
ment  is  in  “technology  planning”.  Valuable  TBMCS  resources  were  utilized  in  the  chase  for  advanced 
distributed  learning  technology.  A  technology  plan  would  have  provided  a  road  map  for  the  implementa¬ 
tion  of  technology  and  would  result  in  more  efficient  expenditure  of  limited  resources  and  budgeting  for 
additional  technology  early  in  the  contract.  The  North  Central  Regional  Technology  in  Education  Con¬ 
sortium  identifies  technology  planning  as  an  ongoing  process  that  translates  organizational  policy,  and 
technology  needs  into  concrete  actions.  Technology  planning  allows  educational  and  training  organiza¬ 
tions  to  take  advantage  of  technology  innovations  while  minimizing  the  negative  impact  of  unexpected 
challenges,  (http://www.netc.org/cdrom/guide/html/gqhome.htm). 

Return  on  Investment.  Smith  (2000)  stated  that  the  DOD  spends  $14  billion  per  year  on  classroom 
education  for  military  and  civilian  personnel.  She  believes  that  distributed  teaching  modules  could  cut 
costs  and  make  it  easier  to  share  training  modules  that  are  applicable  to  numerous  departments.  Webb 
(1999)  stated  that  ROI  is  all  about  accountability  predictions  and  tying  training  to  your  company’s  bottom 
line.  Unfortunately,  there  were  limited  empirical  studies  that  concentrate  on  the  cost  benefits  of  DL  at  the 
conception  of  the  contract.  Hassett  (1992)  identified  costs  in  Computer-Based  Training  (CBT)  as  includ¬ 
ing  developer  salaries  and  benefits  pay,  management  time  required  to  hire  and  supervise  employees,  the 
hours  charged  to  overhead  between  projects,  quality  assurance  reviews,  CBT  maintenance,  and  graphic 
support.  He  also  stated  that  the  average  time  for  creating  an  hour  of  CBT  ranges  from  140  to  316  devel¬ 
oper  hours.  He  quoted  a  study  in  which  61%  of  the  respondents  stated  that  CBT  took  more  than  400  de¬ 
velopment  hours,  27%  stated  it  took  more  than  1,000  development  hours,  and  5%  stated  it  took  4,000 
development  hours  to  develop  a  single  hour  of  CBT.  In  a  review  of  research  of  890  articles  in  distance 
education  from  1990  to  1999  Berge  and  Mrowski  (2001)  reported  only  eight  reflected  cost  data  for  DL. 
Of  the  eight  reports  two  were  written  in  1998,  one  in  1995,  one  in  1994  and  three  in  1991.  The  latter 
studies  were  prior  to  the  advent  of  the  web.  Identifying  the  range  of  possible  costs  for  a  distributed  learn¬ 
ing  training  program  by  utilizing  the  web  as  the  means  for  distribution  was  a  mere  estimation  at  the  time 
of  contract  award  in  1995. 

Policy  and  Management 

1.  Chute,  Thompson,  and  Hancock  (1999)  state  that  people  in  an  organization  fill  three  important 
roles  in  the  change  process:  change  stakeholder,  change  agent,  and  change  target.  The  change 
stakeholder  is  the  individual  or  group  within  the  organizational  power  to  legitimize  the  change, 
the  change  agent  is  the  individual  or  group  responsible  for  implementing  the  change,  and  the 
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change  target  is  the  individual  or  group  whose  knowledge,  skills  attitudes,  or  behaviors  will  be 
altered  as  a  result  of  the  change.  Although  national  level  policy  dictates  an  ADL  concept,  local 
policy  does  not  enforce  distributed  learning.  Change  stakeholders  and  agents  are  clearly  needed 
to  enforce  a  distributed  learning  environment. 

2.  API  13-108  identifies  a  Training  Planning  Team  (TPT)  as  an  action  group  composed  of  repre¬ 
sentatives  from  all  pertinent  functional  areas,  disciplines,  and  interests  involved  in  the  life  cycle 
design,  development,  acquisition,  support,  modification,  funding,  and  management  of  a  specific 
defense  training  system.  The  TPT  uses  the  system  training  plan  to  ensure  training  considerations 
are  adequately  addressed  in  the  defense  system  acquisition  and  modification  processes.  The 
TBMCS  TPT  charter,  dated  1997,  defines  membership  and  tasks.  The  primary  purposes  are  to 
obtain  user  requirements,  ensure  a  sound  structured  approach  to  training  development,  and  en¬ 
sure  all  service  requirements  are  met  as  effectively  and  efficiently  as  possible.  The  TBMCS 
TPT  by  definition  lacks  a  common  DL  vision  and  authority  to  enforce  a  DL  environment. 

Changing  Roles  of  Presentation  Media,  Instructors,  and  Students.  The  widespread  use  of  the  com¬ 
puter  technology  in  all  fields  has  helped  the  virtual  classroom  to  gain  acceptance  among  students,  educa¬ 
tors  and  employers  as  a  means  for  distance  education.  However,  the  concept  of  distributed  learning  has 
far  to  go  before  it  is  accepted  as  equal  to  the  traditional  education  format.  The  change  to  a  virtual  class¬ 
room  requires  a  paradigm  shift  with  upper  management,  students,  and  instmctors.  Evans  (1997)  stated 
that  the  paradigm  shift  includes  issues  such  as  student  authenticity  far  beyond  plagiarism,  the  conver¬ 
gence  of  professional  and  personal  time  produced  by  24-hour,  on-line  teleconferencing  availability,  the 
dependency  on  a  team  of  connectivity  technicians  to  communicate  with  students  and  instmctors,  a  virtual 
higher  education  world,  which  is  less  faculty-centered  and  where  curriculum  is  interdisciplinary.  The  is¬ 
sues  involved  in  this  paradigm  shift  bring  about  challenges,  not  the  least  of  which  is  how  to  develop  a 
military  culture  that  accepts  distance  learning.  TBMCS  focus  groups  revealed  that  students  clearly  did  not 
want  to  learn  solely  from  web-based  instruction,  but  instead  desired  the  assistance  of  the  instructor. 

Design  of  Web-based  Training  Materials.  Cohen  (1999)  revealed  that  generation  X  (bom  between 
1965-1976)  and  generation  Y  (bom  between  1977-1999)  prefer  fast  paced  information  that  flood  them 
with  information  and  want  their  education  to  be  combined  with  entertainment.  These  generations  also 
want  to  review  frequent  feedback  and  a  daily  sense  of  accomplishment.  They  want  latitude  in  when  and 
where  they  study  and  a  choice  of  assignments.  Designing  instmction  to  meet  these  criteria  is  both  time 
consuming  and  costly. 
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9.0— SUMMARY 

The  goals  of  this  paper  were  to  provide:  a)  a  summative  evaluation  that  identifies  strengths,  weak¬ 
nesses,  lessons  learned,  and  best  practices  of  the  TBMCS  training  program,  b)  an  in-depth  analysis  in  as¬ 
sisting  future  C2  decision  makers  in  determining  what  conditions  distributed  learning  is  likely  to  be 
effective  in  for  future  C2  systems,  and  c)  a  holistic  view  (context,  input,  process  and  product  evaluation) 
of  TBMCS  training  that  shows  the  impact  of  training,  not  only  on  the  individual  but  on  the  USAF  as  well. 

9.1  SUMMATIVE  EVALUATION 

A  summative  evaluation  that  identifies  strengths,  weaknesses,  lessons  learned,  and  best  practices  is 
best  summarized  as  follows.  Strengths  are  identified  as  having  a  flexible  contract  and  training  develop¬ 
ment  contractor.  The  TBMCS  procurement  strategy  was  a  cost  plus  contract  with  a  best  effort  clause.  Al¬ 
though  this  acquisition  strategy  resulted  in  considerably  more  risk  on  the  government,  it  allowed  for 
changes  in  scope  as  more  COTS  technology  became  available.  A  fundamental  weakness  was  managing 
the  contract  due  to  the  high  attrition  of  military  personnel  from  Permanent  Change  of  Station  (PCS)  rota¬ 
tions  and  in-house  transfers.  The  Air  Force  does  not  maintain  a  training  Air  Force  Specialty  Code  (AFSC) 
for  officer  personnel.  As  a  result,  most  Air  Force  personnel  lacked  skills  in  applying  the  ISD  process  and 
evaluating  the  various  products.  A  significant  lesson  learned  was  in  the  evaluation  of  the  course.  To  avoid 
controversy  and  scrutiny  from  the  services,  neither  the  developing  contractor  nor  the  office  responsible 
for  managing  the  contract  should  be  in  the  position  to  administer  and  assess  survey  results.  It  would  be 
advantageous  to  all  services  if  an  independent  party  conducted  the  evaluation  of  a  multi-service  training 
program.  Best  practices  are  identified  as  utilizing  the  ISD  process  as  the  basis  to  obtain  requirements  and 
to  design/develop  the  most  cost  effective  and  efficient  training  to  meet  users’  needs.  It  allows  for  user 
validation  of  requirements,  multiple  reviews  of  templates,  prototypes,  demos,  and  end  products,  and  op¬ 
portunities  for  stakeholder  decisions  when  technical  and  cost  trade  offs  are  required. 

9.2  DL  ENVIRONMENT 

Determining  the  best  condition  for  a  distributed  learning  environment  is  challenging.  A  C2  SPD  can 
have  adequate  funding,  the  best  training  materials,  and  deliver  a  quality  product  on  time  to  all  users  in  a 
traditional  training  environment.  However,  there  are  a  myriad  of  obstacles  that  can  contribute  to  the  fail¬ 
ure  of  the  same  training  in  a  distributed  learning  environment.  Prior  to  establishing  a  distributed  learning 
environment  decision  makers  must  do  their  homework.  The  following  questions  are  guides  in  determining 
if  a  supportive  environment  exists.  Negative  responses  can  quickly  change  a  supportive  environment  into 
a  hostile  learning  environment. 
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•  Do  distributed  learning  policies  exist  at  national  and  local  levels? 

•  Does  my  senior  leadership  embrace  a  vision  that  supports  distributed  learning? 

•  Do  I  have  adequate  and  experienced  personnel  to  administer  and  execute  a  training  program  in  a 
distributed  learning  environment? 

•  Does  my  training  contractor  have  experience  in  developing  training  and  administering  distance 
learning  programs? 

•  Do  I  have  control  over  the  training  budget? 

•  What  is  my  commitment  to  a  distributed  learning  initiative  if  my  budget  is  cut? 

•  Does  my  network  infrastructure  support  anytime,  anywhere,  anyplace  learning? 

•  What  are  the  network  bandwidth,  security  constraints,  and  latency  rates  at  the  distributed  loca¬ 
tions? 

•  Does  the  military  culture  support  distributed  learning  environments? 

•  Do  the  training  managers  at  the  distance  locations  have  a  process  in  place  to  support  a  distrib¬ 
uted  learning  environment? 

•  What  organizations  can  I  collaborate  with,  share  lessons  learned  and  best  practices  of  distrib¬ 
uted  learning? 

9.3  HOLISTIC  VIEW  OF  TRAINING 

A  model  for  a  holistic  view  of  training  is  best  described  by  Deming  (2000).  He  identified  a  systems 
theory  as  “a  network  of  interdependent  components  working  together  to  achieve  a  common  aim.”  Figure 
9.3-1  tailors  the  systems  theory  to  the  training  process.  Input  is  defined  as  the  requirements  and  regula¬ 
tions  that  feed  the  system.  The  ISD  and  MTT  process  is  defined  as  the  key  processes  to  training  during 
fielding.  Process  owners  are  defined  as;  AC2ISRC  to  ensure  personnel  receive  Initial  Qualification  Test 
(IQT)  prior  to  arriving  at  their  duty  station,  ESC  training  contractor  to  design,  develop  and  implement 
type-1  training,  and  the  MAJCOMs  to  ensure  processes  are  in  place  for  OJT  and  continuation  training 
after  type-1  training.  Output  is  defined  as  a  qualified  C2  warfighter.  Feedback  is  defined  as  qualitative 
and  quantitative  data  provided  by  students  after  a  course  that  is  used  to  enhance  future  courses.  The  key  to 
the  systems  theory  is  accountability.  All  process  owners  must  complete  their  respective  portion  of  the 
process  in  order  to  support  the  overall  aim  of  the  system.  The  aim  of  the  system  is  defined  as  “a  qualified 
C2  warrior”.  When  process  owners  are  not  accountable,  the  system  becomes  dysfunctional  and  training 
objectives  are  not  met.  Without  a  proper  training  infrastructure  the  system  as  a  whole  cannot  survive. 
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Figure  9.3-1.  Deming  System  Theory  as  it  Applies  to  Training 
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10.0— GLOSSARY 


ADL 

Advanced  Distributed  Learning 

AETC 

Air  Education  Training  Command 

AFDLO 

Air  Force  Distance  Learning  Office 

API 

Air  Force  Instruction 

AFPD 

Air  Force  Policy  Directive 

AFSC 

Air  Force  Specialty  Code 

AOC 

Air  Operations  Center 

APOM 

Amended  Program  Objective  Memorandum 

C2 

Command  and  Control 

CAFC2 

Combat  Air  Force  Command  and  Control 

CBT 

Computer-Based  Training 

CCB 

Configuration  Control  Board 

CCO 

Chief  Combat  Operations 

CIS 

Combat  Intelligence  System 

CFC 

Continuous  Learning  Center 

CONUS 

Continental  United  States 

COTS 

Commercial-Off-the-Shelf 

CMI 

Computer  Managed  Instmction 

CTAPS 

Contingency  Theater  Air  Planning  System 

DAU 

Defense  Acquisition  University 

DCO 

Director  Combat  Operations 

DF 

Distance  Learning 

DOD 

Department  of  Defense 

DT 

Developmental  Test 

EOC 

End  of  Course 

ESC 

Electronic  Systems  Center 

FY 

Fiscal  Year 

LP 

Lesson  Plan 

HTML 

Hypertext  Markup  Language 

HW 

Hardware 

IMI 

Interactive  Multimedia  Instmction 
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IQT 

Initial  Qualification  Test 

ISD 

Instmetio  nal  Systems  Development 

JAO 

Joint  Air  Operations  Center 

JAOTPT 

Joint  Air  Operations  Training  Planning  Team 

JV 

Joint  Vision 

MAJCOM 

Major  Command 

MTT 

Mobile  Training  Team 

NORAD 

North  American  Aerospace  Defense 

OJT 

On  the  Job  Training 

OT 

Operational  Test 

PC 

Personal  Computer 

PE 

Practical  Exercise 

POI 

Program  of  Instmetion 

POM 

Program  Objective  Memorandum 

PSS 

Perimeter  Security  System 

ROI 

Return  On  Investment 

SME 

Subject  Matter  Expert 

SODO 

Senior  Offensive  Duty  Officer 

SOR 

System  of  Record 

SPD 

Systems  Program  Director 

SPO 

System  Program  Office 

SW 

Software 

TADLP 

Total  Army  Distance  Learning  Program 

TASA 

Task  and  Skill  Assessment 

TBMCS 

Theater  Battle  Management  Core  System 

TPR 

Trained  Personnel  Requirement 

TPT 

Training  Planning  Team 

USAE 

United  States  Air  Eorce 

USMC 

United  States  Marine  Corp 

USN 

United  States  Navy 

wees 

Wing  Command  and  Control  System 

WYSIWYG 

What  You  See  Is  What  You  Get 
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APPENDIX  1— END  OF  COURSE  CRITIQUE 

TBMCS  1.0.1  TRAINING 
END  OF  COURSE  CRITIQUE 

Directions: 

Please  answer  all  questions. 

Enter  up  to  50  characters  in  text  entry  fields. 

When  a  field  has  an  arrow  at  the  right,  click  the  arrow  to  display  a  list  of  options 
and  select  one. 

When  answering  a  question  select  the  one  button  that  best  expresses  your 
opinion. 

When  you  have  finished,  press  the  Submit  button  to  register  the  completed  cri¬ 
tique.  You  may  reset  the  form  (clear  it)  by  pressing  the  Reset  button. 

A.  Profile  Data: 

Name  (Last,  First) 


Course  title(select  one) 


Branch  of  Service 


Military  Occupation  position  (MOS/AFSC) 
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Time  at  current  occupation  position  (months) 

1 0  d 

CIS  experience  (months) 

fo  3 

Number  of  operations/ exercises  performed  using  CIS  system. 

I  0  d 

If  more  than  one  operation/  exercise  performed,  the  additional  positions  held 
were: _ 

I  N/A  3 

TBMCS  experience  (months) 

I  0  3 

B.  Training  Evaluation 
B-1.  Presentation  Effectiveness 

The  course  facilitator  ensured  that  training  materials/ equipment  were  ready 
and  operational  before  the  class  started. 

^  strongly  disagree  ^  disagree  ^  agree  ^  strongly  agree 

The  course  facilitator  began  the  class  on  time. 

^  strongly  disagree  ^  disagree  ^  agree  ^  strongly  agree 

The  course  facilitator  was  knowledgeable  about  the  overall  course. 

^  strongly  disagree  ^  disagree  ^  agree  ^  strongly  agree 

The  course  facilitator  clarified  questions  well. 

^  strongly  disagree  ^  disagree  ^  agree  ^  strongly  agree 

The  course  facilitator  gave  relevant  guidance  when  needed. 

^  strongly  disagree  ^  disagree  ^  agree  ^  strongly  agree 

The  course  facilitator  interacted  well  with  the  trainees. 

^  strongly  disagree  ^  disagree  ^  agree  ^  strongly  agree 


68 


Training  Program  Review 
TBMCS  Training 
Program  Evaluation  v.  1.0.1 

B-2  Course  Quality 

Task  completion:  I  believe  that  the  course  provided  the  training  necessary  to 
enable  me  to  complete  the  TBMCS  software  application  tasks  specific  to  my 
work  center. 

^  strongly  disagree  ^  disagree  ^  agree  ^  strongly  agree 

Skill  enhancement:  I  believe  that  the  course  improved  my  ability  to  correctly 
perform  TBMCS  software  application  tasks  when  I  return  to  my  work  center. 

^  strongly  disagree  ^  disagree  ^  agree  ^  strongly  agree 
This  course  was  well  organized. 

^  strongly  disagree  ^  disagree  ^  agree  ^  strongly  agree 

The  level  of  difficulty  of  this  course  was  appropriate. 

^  strongly  disagree  ^  disagree  ^  agree  ^  strongly  agree 

The  length  of  this  course  was  appropriate. 

^  strongly  disagree  ^  disagree  ^  agree  ^  strongly  agree 
B-2.1.  Course  Materials 

This  course  included  appropriate  supporting  materials  (e.g.,  checklists,  CBT, 
interactive  practice,  etc.). 

^  strongly  disagree  ^  disagree  ^  agree  ^  strongly  agree 
The  course  materials  were  useful. 

^  strongly  disagree  ^  disagree  ^  agree  ^  strongly  agree 

B-2.2.  Classroom  Equipment/Environment 

The  reliability  of  the  equipment  used  for  the  course: 

^  Needs  improvement  ^  Was  acceptable 

The  reliability  of  network  connection  for  this  course: 

^  Needs  improvement  ^  Was  acceptable 
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The  classroom  temperature:  (hot,  cold,  humid) 

Needs  improvement  Was  acceptable 

The  classroom  noise  level:  (beepers,  phones,  loud  talking) 

Needs  improvement  Was  acceptable 

The  physical  work  space:(elbow  room) 

Needs  improvement  Was  acceptable 

C.  Overall  Satisfaction 

Overall,  the  course  met  my  expectations. 

^  strongly  disagree  ^  disagree  ^  agree  ^  strongly  agree 

I  believe  I  can  use  this  course  material,  in  its  present  form,  to  train  other  per¬ 
sonnel  when  necessary. 

^  strongly  disagree  ^  disagree  ^  agree  ^  strongly  agree 
Comments: 


Submit 

Reset 
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APPENDIX  2— TBMCS  FORCE  TRAINING  SELF-RATING  FORMS 


ABP/ATO  Planners  Rating  Form 


After  completing  the  exercise,  rate  your  ability  to  accomplish  the  following  tasks  if  you 
were  placed  in  a  real-world  exercise  or  contingency 
Rate  your  performance  using  the  following: 

1 .  If  you  could  not  accomplish  this  task. 

2.  If  you  could  accomplish  the  task  with  some  over-the-shoulder  help. 

3.  If  you  could  accomplish  the  task  with  only  the  on-line  help. 

4.  If  you  could  accomplish  the  task  without  help. 

5.  You  did  not  attempt  to  learn  this  task. 


Using  TAP 

Build  the  following  air  missions _ 

1 .  Ground/Maritime  Target  Missions _ 

2.  RECCE  Missions _ 

3.  Maritime  Target  -  Ground/Maritime  Target  Taskin 

4.  Air  Location  -  Air  Location  Taskin 

5.  [Ground  Alert  -  Ground  Alert  Taskin 

6. 

y.lAETACS  -  AETACS  Taskin 

8.  EC  -  Electronic  Combat  Taskin 

9.  Air  Move  -  Air  Move  Taskin 

10.  Air  Drop  -  Air  Drop  Taskin 

1 1 .  WAG  Missions _ 

12.  Build  escorted  or  packaged  missions _ 

13.  Submit  Refueling,  Close  and  Detached  Escort  Reguests 

14.  Fill  Refueling,  Close  and  Detached  Escert  Requests 

15.  Publish  an  ABP _ 

16. 1  Create  a  new  ABP  from  a  orevious  ABP 


Comments: 


Could  accomplish  without  help 
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ABP/ATO  Replanners  Rating  Form 


After  completing  the  exercise,  rate  your  ability  to  accomplish  the  following  tasks  if  you 

Q. 

Q. 

were  placed  in  a  real-world  exercise  or  contingency 

cb 

0 

0 

0 

Rate  your  performance  using  the  following: 

0 

_C 

3 

O 

1 .  If  you  could  not  accomplish  this  task. 

O 

o 

2.  If  you  could  accomplish  the  task  with  some  over-the-shoulder  help. 

■q. 

£ 

3.  If  you  could  accomplish  the  task  with  only  the  on-line  help. 

o 

o 

o 

■q. 

£  ^ 

■q. 

£ 

■q. 

£ 

TD 

0 

4.  If  you  could  accomplish  the  task  without  help. 

CO 

o 

o  0 
o  -c 

o  s_ 

O 

o 

o 

o 

o 

Q. 

£ 

5.  You  did  not  attempt  to  learn  this  task. 

c 

■D 

CO  0 

■D  P 

CO 

T3 

CO 

■O 

0 

"cO 

3 

O 

Bi 

3 

O 

3 

o 

o 

O 

O  w 

O 

O 

Z 

1 

2 

3 

4 

5 

Using  EMR 


Build  the  following  air  missions 

1. 

Ground/Maritime  Target  Missions 

2. 

RECCE  Missions 

3. 

Maritime  Target  -  Ground/Maritime  Target  Tasking 

4. 

Air  Location  -  Air  Location  Tasking 

5. 

Ground  Alert  -  Ground  Alert  Tasking 

6. 

Tanker  -  Tanker  Tasking 

7. 

AETACS  -  AETACS  Tasking 

8. 

EC  -  Electronic  Combat  Tasking 

9. 

Air  Move  -  Air  Move  Tasking 

10. 

Air  Droo  -  Air  Drop  Tasking 

11. 

Submit  Refueling,  EC  or  Escort  Reguests 

12. 

Publish  an  ATO  Change 

13. 

Replan  single,  unescorted  missions 

14. 

Replan  escorted  or  packaged  missions 

Comments: 
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ABP/ATO  Execution  Monitoring  Rating  Form 


After  compieting  the  exercise,  rate  your  abiiity  to  accompiish  the  foiiowing  tasks  if  you 

Q. 

Q. 

were  piaced  in  a  reai-worid  exercise  or  contingency 

CD 

CD 

CD 

0 

Rate  your  performance  using  the  foiiowing: 

> 

3 

O 

1 .  If  you  couid  not  accompiish  this  task. 

O 

§ 

o 

2.  If  you  could  accomplish  the  task  with  some  over-the-shoulder  help. 

■q. 

E 

CO 

CO 

CO 

3.  If  you  could  accomplish  the  task  with  only  the  on-line  help. 

o 

o 

■q. 

E  ^ 

■q. 

E 

■q. 

E 

TD 

0 

4.  If  you  could  accomplish  the  task  without  help. 

CO 

o 

o  Q> 
o  -C 
CD 

o 

o 

o 

o 

o 

o 

Q. 

E 

5.  You  did  not  attempt  to  learn  this  task. 

c 

p 

CO  CD 
^  1 

CO 

p 

CO 

p 

0 

ti 

CO 

3 

O 

3 

o 

3 

o 

o 

O 

O  w 

O 

O 

Z 

1 

2 

3 

4 

5 

Using  EMC 


Filter  the  ABP  to  look  at  only  specific  missions. 

Interpret  Sortie  Flow  symbology 

Arrange  Sortie  Flow  Table  columns 

Obtain  mission  information  by  rolling  oyer  the  Sortie  Flow  pictograms 

Obtain  mission  information  yia  the  Mission  Status  Filter 

Change  mission  status  from  task  throuoh  mission  comolete 

Enter  mission  deyiations 

Update  runway  statuses 

Using  SAA 


Change  chart  colors 

Zoom,  center  and  pan  chart. 

Plot  selected  airspaces  from  within  SAA. 

Plot  selected  airbases  from  within  SAA. 

Build  an  oyerlay  containing  selected  airspaces  and  bases  from  within  SAA 

Display  selected  oyerlavs. 

Plot  selected  targetes  from  within  SAA. 

Plot  politcal  borders  and  other  Features 

Plot  friendly  missions  from  within  SAA. 
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Airspace  Planners  Rating  Form 


After  completing  the  exercise,  rate  your  ability  to  accomplish  the  following  tasks  if  you 

_9. 

Q. 

were  placed  in  a  real-world  exercise  or  contingency 

<i> 

CD 

CD 

CD 

Rate  your  performance  using  the  following: 

> 

C 

D 

O 

1 .  If  you  could  not  accomplish  this  task. 

o 

§ 

o 

% 

2.  If  you  could  accomplish  the  task  with  some  over-the-shoulder  help. 

■q. 

£ 

CO 

CO 

CO 

3.  If  you  could  accomplish  the  task  with  only  the  on-line  help. 

o 

o 

■q. 

E  ^ 

■q. 

£ 

■q. 

£ 

TD 

O 

4.  If  you  could  accomplish  the  task  without  help. 

CO 

o 

o  Q> 
o  -C 

CJ  s_ 

o 

o 

o 

o 

o 

Q. 

£ 

5.  You  did  not  attempt  to  learn  this  task. 

c 

p 

3 

CO  CD 
?! 

CO 

■O 

3 

CO 

T3 

3 

CD 

tn 

CO 

o 

O 

cS  W 

o 

O 

O 

O 

O 

Z 

1 

2 

3 

4 

5 

Using  AD 


Create  an  ACM 

Modify  an  ACM 

Publish  an  ACO 

Build  filters  to  look  at  airsoaces  for  a  orouo  of  missions,  e.a.  DCA  airsoaces. 

Delete  ACMs 

Copy  ACMs 

Move  location  of  ACMs 

Change  state  of  ACMs 

Shift  time  of  ACMs 
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Reports  Rating  Form 


After  completing  the  exercise,  rate  your  ability  to  accomplish  the  following  tasks  if  you 
were  placed  in  a  real-world  exercise  or  contingency 
Rate  your  performance  using  the  following: 

1 .  If  you  could  not  accomplish  this  task. 

2.  If  you  could  accomplish  the  task  with  some  over-the-shoulder  help. 

3.  If  you  could  accomplish  the  task  with  only  the  on-line  help. 

4.  If  you  could  accomplish  the  task  without  help. 

5.  You  did  not  attempt  to  learn  this  task. 


Reports 


Using  EM  Reports 


1. 

Generate  a  Data  Report  using  a  predefined  template. 

2. 

Generate  a  Detailed  Mission  Data  Report. 

3. 

Print  a  report. 

4. 

Create  and  save  a  Custom  Report  Template. 

5. 

Retrieve  a  saved  Custom  Report  Template  and  use  it  to  generate  a  report. 

6. 

Save  a  EM  Reports  file  in  the  appropriate  format  for  import  into  either  Microsoft 

Open  a  saved  EM  Reports  file  in  either  Microsoft  Excel, 

7. 

Microsoft  Word  or  Microsoft  Access  (if  Office  applications  are  available). 

Comments: 
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User  Alert  Setup  Rating  Form 


After  completing  the  exercise,  rate  your  ability  to  accomplish  the  following  tasks  if  you 
were  placed  in  a  real-world  exercise  or  contingency 
Rate  your  performance  using  the  following: 

1 .  If  you  could  not  accomplish  this  task. 

2.  If  you  could  accomplish  the  task  with  some  over-the-shoulder  help. 

3.  If  you  could  accomplish  the  task  with  only  the  on-line  help. 

4.  If  you  could  accomplish  the  task  without  help. 

5.  You  did  not  attempt  to  learn  this  task. 


1. 


Explain  the  importance  of  building  an  alert  distribution  matrix  to  your  senior 
leadership  (CCP,  CCO,  SWO  or  others). _ 


Using  DLM _ 

Build  distribution  lists  for  global/internal  addressees  (profilename). _ 

Build  distribution  lists  for  local/internal  addressees  (localprofilename@hostname 

Build  distribution  lists  for  any  of  the  nine  reserved  alerts. _ 

View  and  edit  a  distribution  list. _ 


Using  User  Alerts _ 

6.  Ilest  distribution  lists  built  in  DLM. 


Using  EMR _ 

Build  an  air  mission  alert  (CNX/DEL/ABT/DIV,  MISSION  LOST,  ACK/ORD/ACR 

Build  distribution  lists  for  alerts  other  than  air  missions. _ 

Start  Alert  Monitoring _ 

10.  Subscribe  to  an  ABP. _ 

1 1 .  Interrupt  ABP  monitoring  to  make  a  database  change  (enter  Setup  Mode). 


Comments: 


